Sep 09 21:31:07.895517 ip-10-0-128-94 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 21:31:07.895527 ip-10-0-128-94 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 21:31:07.895534 ip-10-0-128-94 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 21:31:07.895676 ip-10-0-128-94 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 21:31:18.027642 ip-10-0-128-94 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 21:31:18.027665 ip-10-0-128-94 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 718dd77aa2cb4e0ca8337ebc287f3b47 --
Sep 09 21:33:16.034538 ip-10-0-128-94 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 21:33:16.396786 ip-10-0-128-94 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:33:16.396786 ip-10-0-128-94 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 21:33:16.396786 ip-10-0-128-94 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:33:16.396786 ip-10-0-128-94 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 21:33:16.396786 ip-10-0-128-94 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:33:16.398362 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.398280    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 21:33:16.404688 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404668    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:16.404688 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404685    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:16.404688 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404689    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:16.404688 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404692    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:16.404688 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404695    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404698    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404701    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404704    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404707    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404709    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404712    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404715    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404717    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404720    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404722    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404725    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404727    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404731    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404733    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404736    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404738    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404741    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404743    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404746    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:16.404870 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404748    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404751    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404753    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404756    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404758    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404761    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404764    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404766    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404769    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404773    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404777    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404780    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404783    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404785    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404788    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404790    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404793    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404795    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404799    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:16.405366 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404801    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404804    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404806    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404809    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404811    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404813    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404816    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404819    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404822    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404824    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404827    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404829    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404832    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404835    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404837    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404840    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404842    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404845    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404847    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404850    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:16.405815 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404852    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404855    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404858    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404862    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404866    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404869    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404872    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404876    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404879    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404882    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404885    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404887    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404890    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404893    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404896    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404898    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404901    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404903    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404906    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:16.406322 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404908    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404911    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404914    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404916    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404919    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404921    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404924    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404926    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404928    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404931    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404933    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404936    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404939    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404941    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404946    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404950    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.404953    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406411    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406418    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:16.406769 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406422    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406425    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406428    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406430    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406433    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406436    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406439    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406441    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406444    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406447    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406449    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406452    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406454    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406457    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406459    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406462    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406465    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406467    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406479    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406481    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:16.407217 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406484    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406487    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406490    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406492    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406496    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406499    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406502    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406504    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406507    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406510    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406513    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406515    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406518    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406521    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406523    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406526    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406528    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406531    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406535    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:16.407726 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406538    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406540    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406543    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406545    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406548    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406550    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406553    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406555    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406558    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406561    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406563    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406566    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406568    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406570    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406573    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406575    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406578    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406581    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406583    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406588    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:16.408183 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406592    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406595    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406598    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406601    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406603    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406606    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406609    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406611    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406615    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406619    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406622    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406624    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406627    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406630    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406633    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406635    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406638    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406641    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406643    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:16.408696 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406646    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406649    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406651    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406654    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406657    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406659    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406662    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406665    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406667    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406669    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406673    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406675    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406678    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406681    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406683    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406686    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406689    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406692    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.406694    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407061    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:16.409158 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407065    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407068    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407071    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407074    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407076    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407079    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407082    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407084    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407087    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407091    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407095    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407099    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407102    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407105    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407108    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407111    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407114    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407116    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407120    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:16.409663 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407124    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407126    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407129    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407132    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407134    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407137    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407140    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407143    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407147    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407150    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407152    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407155    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407157    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407160    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407162    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407165    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407167    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407170    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407172    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:16.410130 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407175    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407179    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407182    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407185    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407188    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407190    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407193    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407196    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407199    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407201    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407204    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407207    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407210    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407212    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407214    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407217    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407220    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407223    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407226    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:16.410626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407229    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407233    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407235    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407238    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407240    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407243    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407245    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407248    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407250    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407253    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407255    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407258    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407260    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407263    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407280    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407283    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407286    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407288    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407291    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407293    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:16.411084 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407296    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407299    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407301    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407304    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407307    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407310    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407312    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407315    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407319    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407322    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407324    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407328    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407331    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407333    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407336    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407338    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407340    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407343    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407345    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407348    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:16.411586 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.407351    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407424    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407456    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407462    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407467    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407470    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407473    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407477    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407482    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407485    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407487    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407491    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407494    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407497    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407500    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407503    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407506    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407513    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407516    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407520    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407524    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407527    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407530    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407533    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 21:33:16.412090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407541    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407544    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407547    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407550    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407553    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407556    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407559    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407562    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407565    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407569    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407572    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407574    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407578    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407581    2585 flags.go:64] FLAG: --enable-server="true"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407583    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407588    2585 flags.go:64] FLAG: --event-burst="100"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407591    2585 flags.go:64] FLAG: --event-qps="50"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407593    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407596    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407599    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407602    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407605    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407608    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407611    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407614    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 21:33:16.412694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407617    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407620    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407623    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407627    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407630    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407633    2585 flags.go:64] FLAG: --feature-gates=""
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407636    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407639    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407642    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407645    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407648    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407651    2585 flags.go:64] FLAG: --help="false"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407654    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407657    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407659    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407662    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407665    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407669    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407671    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407674    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407677    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407679    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407682    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407685    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 21:33:16.413352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407688    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407691    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407693    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407696    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407699    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407701    2585 flags.go:64] FLAG: --lock-file=""
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407704    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407707    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407710    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407715    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407718    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407722    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407725    2585 flags.go:64] FLAG: --logging-format="text"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407728    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407731    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407734    2585 flags.go:64] FLAG: --manifest-url=""
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407737    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407741    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407744    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407748    2585 flags.go:64] FLAG: --max-pods="110"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407751    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407754    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407756    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407759    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407762    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 21:33:16.414037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407765    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407768    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407774    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407777    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407780    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407783    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407786    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407791    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407794    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407797    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407799    2585 flags.go:64] FLAG: --port="10250"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407802    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407805    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ab2407d4d5ae1f3a"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407808    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407811    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407813    2585 flags.go:64] FLAG: --register-node="true"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407816    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407821    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407825    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407828    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407831    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407834    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407837    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407840    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407843    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 21:33:16.414671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407845    2585 flags.go:64] FLAG: --runonce="false"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407848    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407851    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407854    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407857    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407860    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407862    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407865    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407868    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407871    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407874    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407877    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407880    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407883    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407885    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407888    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407893    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407896    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407898    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407902    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407904    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407907    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407922    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407926    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407931    2585 flags.go:64] FLAG: --v="2"
Sep 09 21:33:16.415515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407935    2585 flags.go:64] FLAG: --version="false"
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407939    2585 flags.go:64] FLAG: --vmodule=""
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407943    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.407946    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408052    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408055    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408059    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408061    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408064    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408067    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408069    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408072    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408075    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408077    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408080    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408082    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408085    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408087    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408090    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408093    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408095    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:16.416594 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408098    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408101    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408103    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408106    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408108    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408111    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408113    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408116    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408118    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408121    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408125    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408128    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408131    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408133    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408136    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408140    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408143    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408147    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408150    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:16.417504 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408153    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408156    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408158    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408161    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408163    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408166    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408168    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408171    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408173    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408176    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408178    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408181    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408183    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408186    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408189    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408192    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408196    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408199    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408202    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408205    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:16.418010 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408207    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408210    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408212    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408216    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408219    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408221    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408224    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408227    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408230    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408232    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408235    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408237    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408240    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408242    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408245    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408247    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408250    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408253    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408255    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:16.418782 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408258    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408260    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408263    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408281    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408284    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408286    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408289    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408291    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408295    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408298    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408300    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408303    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408305    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408308    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408311    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408313    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408317    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408321    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408324    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:16.419617 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408326    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408329    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408332    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408334    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.408337    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.409062    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.417350    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.417370    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.418190    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.418204    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418309    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418318    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418322    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418327    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418331    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:16.420142 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418336    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418340    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418345    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418350    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418354    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418358    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418363    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418367    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418372    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418376    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418380    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418384    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418388    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418392    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418396    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418400    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418404    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418408    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418413    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418417    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:16.420724 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418421    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418425    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418429    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418433    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418438    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418443    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418449    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418455    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418460    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418464    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418468    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418472    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418476    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418480    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418484    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418488    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418492    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418496    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418500    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:16.421234 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418504    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418508    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418512    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418517    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418521    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418524    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418529    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418533    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418537    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418541    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418545    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418549    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418554    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418558    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418562    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418566    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418570    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418576    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418581    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418585    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:16.421862 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418589    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418593    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418597    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418601    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418606    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418609    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418613    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418620    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418625    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418630    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418634    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418639    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418643    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418647    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418651    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418655    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418661    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418668    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418672    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:16.422820 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418677    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418681    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418685    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418689    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418693    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418697    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418701    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418705    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418709    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418714    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418718    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418723    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418728    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418732    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418736    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:16.423294 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418741    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.418749    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418965    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418975    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418980    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418985    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418990    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418994    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.418999    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419004    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419009    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419013    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419017    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419021    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419025    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:16.423807 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419031    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419036    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419040    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419045    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419049    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419053    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419057    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419061    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419065    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419069    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419073    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419076    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419082    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419086    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419090    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419102    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419106    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419110    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419114    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419118    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:16.424178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419122    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419126    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419130    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419134    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419138    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419141    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419145    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419149    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419153    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419157    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419161    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419166    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419169    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419173    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419177    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419181    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419185    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419189    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419193    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419197    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:16.424745 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419201    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419205    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419209    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419213    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419217    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419221    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419226    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419232    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419245    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419250    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419255    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419259    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419263    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419286    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419291    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419296    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419300    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419304    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419308    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:16.425297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419312    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419316    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419320    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419324    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419328    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419332    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419336    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419340    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419344    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419348    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419352    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419357    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419362    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419366    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419370    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419373    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419377    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419385    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419390    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:16.425781 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419394    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419398    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419403    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419415    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419420    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419424    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419428    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:16.419432    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.419440    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.419596    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.423426    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.424421    2585 server.go:1019] "Starting client certificate rotation"
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.424531    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 21:33:16.426297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.424567    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 21:33:16.428208 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.428192    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 21:33:16.429572 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.429554    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 21:33:16.442634 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.442619    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 09 21:33:16.446325 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.446310    2585 log.go:25] "Validated CRI v1 image API"
Sep 09 21:33:16.448582 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.448561    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 21:33:16.451715 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.451698    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 21:33:16.451921 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.451905    2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bd072645-104c-4bf8-88a7-e4d7e553fb88:/dev/nvme0n1p3 e959e74b-4291-45bc-9070-5372ac09728a:/dev/nvme0n1p4]
Sep 09 21:33:16.451965 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.451922    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 21:33:16.456889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.456788    2585 manager.go:216] Machine: {Timestamp:2026-09-09 21:33:16.455522374 +0000 UTC m=+0.310783512 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3135688 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec213014d38f1cf6b2e4acd7af1a9520 SystemUUID:ec213014-d38f-1cf6-b2e4-acd7af1a9520 BootID:718dd77a-a2cb-4e0c-a833-7ebc287f3b47 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:b4:e2:f1:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:b4:e2:f1:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:fb:33:c4:9c:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 21:33:16.456889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.456884    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 21:33:16.457002 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.456954    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 21:33:16.460204 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.460183    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 21:33:16.460346 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.460206    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 21:33:16.461078 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.461068    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 21:33:16.461111 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.461079    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 21:33:16.461111 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.461087    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 21:33:16.461823 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.461814    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 21:33:16.463453 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.463443    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 21:33:16.467333 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467322    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 21:33:16.467441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467432    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 21:33:16.467471 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467460    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 21:33:16.467471 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467469    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 21:33:16.467526 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467482    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 21:33:16.467526 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467491    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 09 21:33:16.467526 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467499    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 21:33:16.467912 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467901    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 21:33:16.467948 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.467921    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 21:33:16.470020 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.470001    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 21:33:16.472959 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.472943    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 21:33:16.473005 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.472976    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 21:33:16.473133 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473122    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 21:33:16.473164 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473141    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 21:33:16.473164 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473147    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 21:33:16.473164 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473153    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 21:33:16.473164 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473160    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 21:33:16.473164 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473165    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 21:33:16.473665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473657    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 21:33:16.473700 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473666    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 21:33:16.473700 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473672    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 21:33:16.473700 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473684    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 21:33:16.473700 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473696    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 21:33:16.473801 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.473704    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 21:33:16.474252 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.474242    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 21:33:16.474252 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.474252    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 21:33:16.476633 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.476610    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p9pfl"
Sep 09 21:33:16.478051 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.478025    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 21:33:16.478161 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.478146    2585 server.go:1268] "Started kubelet"
Sep 09 21:33:16.478223 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.478200    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 21:33:16.478297 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.478224    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 21:33:16.478352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.478257    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 21:33:16.478424 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.478399    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 21:33:16.480156 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.480136    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 21:33:16.480411 ip-10-0-128-94 systemd[1]: Started Kubernetes Kubelet.
Sep 09 21:33:16.481885 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.481738    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 21:33:16.482214 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.482191    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 21:33:16.483552 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.483026    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 21:33:16.483842 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.483827    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p9pfl"
Sep 09 21:33:16.485657 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.484947    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-94.ec2.internal.18d3c48f3cee0e09  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-94.ec2.internal,UID:ip-10-0-128-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-94.ec2.internal,},FirstTimestamp:2026-09-09 21:33:16.477820425 +0000 UTC m=+0.333081563,LastTimestamp:2026-09-09 21:33:16.477820425 +0000 UTC m=+0.333081563,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-94.ec2.internal,}"
Sep 09 21:33:16.488632 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.488614    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 21:33:16.488713 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.488642    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 21:33:16.488772 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.488735    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:16.489359 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.489301    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 21:33:16.489585 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.489572    2585 factory.go:55] Registering systemd factory
Sep 09 21:33:16.489620 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.489588    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 09 21:33:16.492889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.492864    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 21:33:16.492889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.492890    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 21:33:16.493043 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.492997    2585 factory.go:153] Registering CRI-O factory
Sep 09 21:33:16.493043 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.493006    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 21:33:16.493043 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.493013    2585 factory.go:223] Registration of the crio container factory successfully
Sep 09 21:33:16.493169 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.493078    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 21:33:16.493169 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.493096    2585 factory.go:103] Registering Raw factory
Sep 09 21:33:16.493169 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.493111    2585 manager.go:1196] Started watching for new ooms in manager
Sep 09 21:33:16.493330 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.493015    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 21:33:16.493519 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.493502    2585 manager.go:349] Starting recovery of all containers
Sep 09 21:33:16.499753 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.499731    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:16.501455 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.501440    2585 manager.go:354] Recovery completed
Sep 09 21:33:16.505471 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.505457    2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-94.ec2.internal\" not found" node="ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.506786 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.506770    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:16.510457 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.510385    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:16.510457 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.510407    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:16.510457 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.510420    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:16.511802 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.511783    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 21:33:16.511875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.511802    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 21:33:16.511875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.511813    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:16.511972 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.511829    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 21:33:16.513174 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.513160    2585 policy_none.go:49] "None policy: Start"
Sep 09 21:33:16.513233 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.513186    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 21:33:16.513233 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.513198    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 21:33:16.513309 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.513159    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:16.513309 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.513287    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:16.513309 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.513301    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:16.514342 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.514332    2585 policy_none.go:47] "Start"
Sep 09 21:33:16.558824 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.558809    2585 manager.go:337] "Starting Device Plugin manager"
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.558844    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.558853    2585 server.go:87] "Starting device plugin registration server"
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.559067    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.559078    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.559177    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.559256    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.559281    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.559677    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 21:33:16.566009 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.559725    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:16.638756 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.638626    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 21:33:16.640030 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.640013    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 21:33:16.640094 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.640074    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 21:33:16.640138 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.640105    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 21:33:16.640184 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.640151    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 21:33:16.642692 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.642672    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:16.660664 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.660617    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:16.661665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.661652    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:16.661735 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.661679    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:16.661735 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.661694    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:16.661735 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.661718    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.669705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.669693    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.669747 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.669713    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-94.ec2.internal\": node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:16.682060 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.682042    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:16.740632 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.740599    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"]
Sep 09 21:33:16.740735 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.740674    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:16.742317 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.742303    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:16.742400 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.742328    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:16.742400 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.742338    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:16.743606 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.743595    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:16.743695 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.743681    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.743739 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.743709    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:16.744394 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.744377    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:16.744478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.744400    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:16.744478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.744408    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:16.744478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.744418    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:16.744478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.744422    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:16.744478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.744432    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:16.746276 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.746254    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.746328 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.746293    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:16.747056 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.747039    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:16.747104 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.747081    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:16.747104 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.747099    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:16.759801 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.759783    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-94.ec2.internal\" not found" node="ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.763730 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.763715    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-94.ec2.internal\" not found" node="ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.782591 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.782573    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:16.794788 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.794769    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2fae258013d512334aa1ba0a14e3b988-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"2fae258013d512334aa1ba0a14e3b988\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.794834 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.794793    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fae258013d512334aa1ba0a14e3b988-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"2fae258013d512334aa1ba0a14e3b988\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.794834 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.794811    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcde823c27ff5664b9e4fd7c2bf072bf-config\") pod \"kube-apiserver-proxy-ip-10-0-128-94.ec2.internal\" (UID: \"fcde823c27ff5664b9e4fd7c2bf072bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.883041 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.883020    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:16.895415 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.895399    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2fae258013d512334aa1ba0a14e3b988-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"2fae258013d512334aa1ba0a14e3b988\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.895466 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.895421    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fae258013d512334aa1ba0a14e3b988-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"2fae258013d512334aa1ba0a14e3b988\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.895466 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.895440    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcde823c27ff5664b9e4fd7c2bf072bf-config\") pod \"kube-apiserver-proxy-ip-10-0-128-94.ec2.internal\" (UID: \"fcde823c27ff5664b9e4fd7c2bf072bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.895526 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.895476    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcde823c27ff5664b9e4fd7c2bf072bf-config\") pod \"kube-apiserver-proxy-ip-10-0-128-94.ec2.internal\" (UID: \"fcde823c27ff5664b9e4fd7c2bf072bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.895526 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.895515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fae258013d512334aa1ba0a14e3b988-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"2fae258013d512334aa1ba0a14e3b988\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.895584 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:16.895515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2fae258013d512334aa1ba0a14e3b988-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"2fae258013d512334aa1ba0a14e3b988\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:16.983895 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:16.983825    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:17.064375 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.064347    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:17.067357 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.067331    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:17.084066 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:17.084043    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:17.184609 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:17.184569    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:17.285128 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:17.285048    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:17.385623 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:17.385597    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:17.424042 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.424016    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 21:33:17.424445 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.424165    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 21:33:17.424445 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.424165    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 21:33:17.486479 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:17.486455    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:17.489263 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.489235    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 21:28:17 +0000 UTC" deadline="2028-06-11 06:04:21.709448284 +0000 UTC"
Sep 09 21:33:17.489329 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.489265    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15368h31m4.220186724s"
Sep 09 21:33:17.581114 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.581048    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:17.586807 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:17.586790    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:17.687485 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:17.687461    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Sep 09 21:33:17.743736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.743714    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:17.763986 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:17.763959    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fae258013d512334aa1ba0a14e3b988.slice/crio-0872aac5b9df57267be97ef8d96578ad582df115c447f78a664e61171aa0a67d WatchSource:0}: Error finding container 0872aac5b9df57267be97ef8d96578ad582df115c447f78a664e61171aa0a67d: Status 404 returned error can't find the container with id 0872aac5b9df57267be97ef8d96578ad582df115c447f78a664e61171aa0a67d
Sep 09 21:33:17.764536 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:17.764498    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcde823c27ff5664b9e4fd7c2bf072bf.slice/crio-c83cde5926869b1acce8103758a5ad6577d0b62593570e0f4c139afac05d2d6a WatchSource:0}: Error finding container c83cde5926869b1acce8103758a5ad6577d0b62593570e0f4c139afac05d2d6a: Status 404 returned error can't find the container with id c83cde5926869b1acce8103758a5ad6577d0b62593570e0f4c139afac05d2d6a
Sep 09 21:33:17.768584 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.768566    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:33:17.789548 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.789531    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:17.799102 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.799085    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 21:33:17.800087 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.800075    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Sep 09 21:33:17.807956 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:17.807943    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 21:33:19.949581 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.949509    2585 apiserver.go:52] "Watching apiserver"
Sep 09 21:33:19.956889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.955969    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 21:33:19.956889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.956850    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal","openshift-multus/multus-additional-cni-plugins-t4dct","openshift-multus/multus-b9qzc","openshift-multus/network-metrics-daemon-qvhdr","kube-system/konnectivity-agent-hsd9r","openshift-cluster-node-tuning-operator/tuned-vv7b4","openshift-dns/node-resolver-rnh76","openshift-image-registry/node-ca-k5kh5","openshift-network-diagnostics/network-check-target-g6bm5","openshift-network-operator/iptables-alerter-t2qnc","openshift-ovn-kubernetes/ovnkube-node-w5txw","kube-system/global-pull-secret-syncer-mgvxv"]
Sep 09 21:33:19.958565 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.958538    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:19.961085 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.961062    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.961676 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.961657    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 21:33:19.961775 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.961682    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9zwlx\""
Sep 09 21:33:19.961991 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.961976    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 21:33:19.962462 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.962412    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:19.962559 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:19.962500    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:19.963946 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.963931    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:19.964381 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.964361    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 21:33:19.964653 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.964507    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mxj4t\""
Sep 09 21:33:19.964653 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.964626    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 21:33:19.964653 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.964632    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 21:33:19.965089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.964849    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.965089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.965076    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 21:33:19.965335 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.965279    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 21:33:19.965613 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.965596    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 21:33:19.966227 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.965995    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 21:33:19.966227 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.966067    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 21:33:19.966382 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.966340    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8cbxc\""
Sep 09 21:33:19.966816 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.966448    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 21:33:19.966816 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.966630    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 21:33:19.966816 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.966687    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 21:33:19.967049 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.966911    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 21:33:19.967111 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.967055    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qcxtt\""
Sep 09 21:33:19.967709 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.967688    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:19.967814 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:19.967773    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:19.968943 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.968907    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:19.970484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.970459    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 21:33:19.971397 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.971379    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 21:33:19.971648 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.971633    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dwrml\""
Sep 09 21:33:19.971873 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.971858    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 21:33:19.972972 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.972955    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.975434 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.975400    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:19.977534 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.976895    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:19.977996 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.977868    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6cn4p\""
Sep 09 21:33:19.977996 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.977928    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5f9ff\""
Sep 09 21:33:19.978193 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.978178    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 21:33:19.978296 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.978220    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:33:19.978403 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.978381    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 21:33:19.978468 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.978427    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 21:33:19.978639 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.978623    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 21:33:19.979094 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.979076    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 21:33:19.979322 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.979295    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:19.979403 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:19.979359    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:19.979664 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.979642    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kfp2c\""
Sep 09 21:33:19.979777 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.979756    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 21:33:19.979873 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.979853    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 21:33:19.980476 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.980460    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:19.982375 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982351    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-var-lib-kubelet\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.982481 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982397    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-log-socket\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.982481 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982424    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cnibin\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:19.982481 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982446    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-slash\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.982481 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982472    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-run-netns\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982498    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-system-cni-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982530    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-k8s-cni-cncf-io\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982557    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-cni-multus\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982580    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-etc-kubernetes\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982607    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-run\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982628    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-systemd\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982650    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982674    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-env-overrides\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982696    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982731    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-node-log\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.982759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982753    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:19.983315 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982786    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:19.983315 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982812    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:19.983315 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.983266    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-daemon-config\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.983465 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.983356    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfbj\" (UniqueName: \"kubernetes.io/projected/b736c890-2a7a-4bfa-8336-0add1879d1bb-kube-api-access-dmfbj\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.983465 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.983389    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.983465 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.983420    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:19.983616 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.983487    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-cni-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.983616 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.983514    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-conf-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.983827 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.982450    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:33:19.985117 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985100    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 21:33:19.985429 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985414    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 21:33:19.985738 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985680    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovnkube-script-lib\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.985810 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985762    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7jf\" (UniqueName: \"kubernetes.io/projected/257e50ad-c41f-420c-b697-a1ea0d80bb2d-kube-api-access-bc7jf\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.985810 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985786    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-cnibin\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.985902 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985813    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1395e458-dd6f-4c0c-9fdd-bd31989597de-cni-binary-copy\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.985902 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985841    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysctl-conf\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.985902 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:19.985902 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985896    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02af621e-6be7-4290-982f-8b6ee14bfd8c-konnectivity-ca\") pod \"konnectivity-agent-hsd9r\" (UID: \"02af621e-6be7-4290-982f-8b6ee14bfd8c\") " pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:19.986074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985919    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-ovn\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985948    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-modprobe-d\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.986074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.985976    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-system-cni-dir\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:19.986074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986002    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2017749-1a67-4077-8ae7-126c4fa84cf2-kubelet-config\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:19.986074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-hostroot\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.986074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986057    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysconfig\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-host\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986111    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-systemd-units\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986138    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-cni-bin\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986167    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-socket-dir-parent\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986193    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-tuned\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986240    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-cni-bin\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986267    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b736c890-2a7a-4bfa-8336-0add1879d1bb-tmp\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986292    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-cni-netd\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986315    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfj6\" (UniqueName: \"kubernetes.io/projected/b4059f83-eb9e-4c96-b934-0fb98b2716da-kube-api-access-wrfj6\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:19.986352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986345    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysctl-d\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986373    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02af621e-6be7-4290-982f-8b6ee14bfd8c-agent-certs\") pod \"konnectivity-agent-hsd9r\" (UID: \"02af621e-6be7-4290-982f-8b6ee14bfd8c\") " pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986400    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-var-lib-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986428    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b875a80b-0dd4-4af6-8222-2f86c18b50a8-hosts-file\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986461    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-kubelet\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986489    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovnkube-config\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986519    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzs78\" (UniqueName: \"kubernetes.io/projected/03550965-bdc0-4bfe-8915-b27d8e65b0d8-kube-api-access-mzs78\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986549    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b875a80b-0dd4-4af6-8222-2f86c18b50a8-tmp-dir\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986573    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-systemd\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986599    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-sys\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986627    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-etc-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986654    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovn-node-metrics-cert\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986714    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2017749-1a67-4077-8ae7-126c4fa84cf2-dbus\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986743    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrxb\" (UniqueName: \"kubernetes.io/projected/1395e458-dd6f-4c0c-9fdd-bd31989597de-kube-api-access-zbrxb\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.986779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986771    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-kubernetes\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.987494 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986800    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-lib-modules\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:19.987494 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986827    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkdj\" (UniqueName: \"kubernetes.io/projected/b875a80b-0dd4-4af6-8222-2f86c18b50a8-kube-api-access-5gkdj\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:19.987494 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986855    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-netns\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.987494 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986885    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-kubelet\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.987494 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.986909    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-multus-certs\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:19.989092 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.989052    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kv8vh\""
Sep 09 21:33:19.994174 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:19.994153    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 21:33:20.005329 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.005309    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w9dwd"
Sep 09 21:33:20.014196 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.014180    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w9dwd"
Sep 09 21:33:20.071200 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.071174    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 21:33:20.088043 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088017    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-modprobe-d\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.088160 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-system-cni-dir\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.088160 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088081    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2017749-1a67-4077-8ae7-126c4fa84cf2-kubelet-config\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:20.088160 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088101    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-hostroot\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.088160 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2kz\" (UniqueName: \"kubernetes.io/projected/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-kube-api-access-pn2kz\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.088160 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088144    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysconfig\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088164    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-host\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088184    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-modprobe-d\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-systemd-units\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088232    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-systemd-units\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-cni-bin\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088278    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-socket-dir-parent\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088285    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-system-cni-dir\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088304    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088332    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2017749-1a67-4077-8ae7-126c4fa84cf2-kubelet-config\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088332    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-tuned\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088364    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-cni-bin\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-device-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088413    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2708ff9c-c567-436c-a587-54c9ec3213b7-host-slash\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.088443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088434    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b736c890-2a7a-4bfa-8336-0add1879d1bb-tmp\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088456    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-cni-netd\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088480    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfj6\" (UniqueName: \"kubernetes.io/projected/b4059f83-eb9e-4c96-b934-0fb98b2716da-kube-api-access-wrfj6\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088505    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysctl-d\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02af621e-6be7-4290-982f-8b6ee14bfd8c-agent-certs\") pod \"konnectivity-agent-hsd9r\" (UID: \"02af621e-6be7-4290-982f-8b6ee14bfd8c\") " pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088548    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-var-lib-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088570    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b875a80b-0dd4-4af6-8222-2f86c18b50a8-hosts-file\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088605    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088628    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8651d356-a086-46cb-9f76-940a0cee764a-serviceca\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088666    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-kubelet\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088684    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovnkube-config\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088701    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzs78\" (UniqueName: \"kubernetes.io/projected/03550965-bdc0-4bfe-8915-b27d8e65b0d8-kube-api-access-mzs78\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088718    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b875a80b-0dd4-4af6-8222-2f86c18b50a8-tmp-dir\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088738    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-registration-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-systemd\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088789    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-sys\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088804    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-etc-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088823    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.089175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088856    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovn-node-metrics-cert\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088878    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2017749-1a67-4077-8ae7-126c4fa84cf2-dbus\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088916    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrxb\" (UniqueName: \"kubernetes.io/projected/1395e458-dd6f-4c0c-9fdd-bd31989597de-kube-api-access-zbrxb\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088939    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-kubernetes\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088946    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-socket-dir-parent\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088963    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-lib-modules\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.088964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysctl-d\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089002    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-cni-netd\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-hostroot\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089038    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkdj\" (UniqueName: \"kubernetes.io/projected/b875a80b-0dd4-4af6-8222-2f86c18b50a8-kube-api-access-5gkdj\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-netns\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089093    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-kubelet\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089104    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysconfig\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089122    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-multus-certs\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089155    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-host\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089168    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8651d356-a086-46cb-9f76-940a0cee764a-host\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089195    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-var-lib-kubelet\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089221    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-cni-bin\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.089800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089237    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-log-socket\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089261    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cnibin\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089280    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b875a80b-0dd4-4af6-8222-2f86c18b50a8-hosts-file\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089286    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk44m\" (UniqueName: \"kubernetes.io/projected/8651d356-a086-46cb-9f76-940a0cee764a-kube-api-access-nk44m\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089292    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-cni-bin\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089315    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-slash\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089403    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-var-lib-kubelet\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089421    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-multus-certs\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-var-lib-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089479    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-run-netns\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089501    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-system-cni-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-k8s-cni-cncf-io\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089543    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-cni-multus\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089563    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2017749-1a67-4077-8ae7-126c4fa84cf2-dbus\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089586    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-cni-multus\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089599    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-netns\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089615    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-systemd\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089617    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-slash\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.090705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089632    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-var-lib-kubelet\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089654    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-log-socket\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089705    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-sys\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089947    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b875a80b-0dd4-4af6-8222-2f86c18b50a8-tmp-dir\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.089995    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cnibin\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090026    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-etc-kubernetes\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090053    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090080    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqhw2\" (UniqueName: \"kubernetes.io/projected/2708ff9c-c567-436c-a587-54c9ec3213b7-kube-api-access-vqhw2\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-run\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090146    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-systemd\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090189    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-lib-modules\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090195    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-env-overrides\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090242    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090247    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-etc-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090271    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090293    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-host-run-k8s-cni-cncf-io\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.101875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-node-log\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090298    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-system-cni-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090120    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-run-netns\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090363    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090376    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-kubernetes\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090390    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090416    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090440    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-daemon-config\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090467    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2708ff9c-c567-436c-a587-54c9ec3213b7-iptables-alerter-script\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090496    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfbj\" (UniqueName: \"kubernetes.io/projected/b736c890-2a7a-4bfa-8336-0add1879d1bb-kube-api-access-dmfbj\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090547    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090573    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-cni-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090599    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-conf-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090623    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-sys-fs\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090742    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovnkube-script-lib\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090770    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7jf\" (UniqueName: \"kubernetes.io/projected/257e50ad-c41f-420c-b697-a1ea0d80bb2d-kube-api-access-bc7jf\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.102742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090798    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-cnibin\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090825    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1395e458-dd6f-4c0c-9fdd-bd31989597de-cni-binary-copy\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090836    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090852    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-socket-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090876    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysctl-conf\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090926    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02af621e-6be7-4290-982f-8b6ee14bfd8c-konnectivity-ca\") pod \"konnectivity-agent-hsd9r\" (UID: \"02af621e-6be7-4290-982f-8b6ee14bfd8c\") " pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090927    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-env-overrides\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.090952    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-ovn\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.091024    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091032    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-ovn\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.091099    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs podName:b4059f83-eb9e-4c96-b934-0fb98b2716da nodeName:}" failed. No retries permitted until 2026-09-09 21:33:20.591065239 +0000 UTC m=+2.965389074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs") pod "network-metrics-daemon-qvhdr" (UID: "b4059f83-eb9e-4c96-b934-0fb98b2716da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091202    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-systemd\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091259    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-etc-kubernetes\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091318    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-run\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091352    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-node-log\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-cnibin\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.103656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovnkube-script-lib\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091840    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovnkube-config\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091889    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-kubelet\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091886    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1395e458-dd6f-4c0c-9fdd-bd31989597de-cni-binary-copy\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091935    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-run-openvswitch\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.091962    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-daemon-config\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092019    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03550965-bdc0-4bfe-8915-b27d8e65b0d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092026    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-sysctl-conf\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092061    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092477    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/257e50ad-c41f-420c-b697-a1ea0d80bb2d-ovn-node-metrics-cert\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092492    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257e50ad-c41f-420c-b697-a1ea0d80bb2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092537    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092560    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-conf-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.092570    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092571    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02af621e-6be7-4290-982f-8b6ee14bfd8c-konnectivity-ca\") pod \"konnectivity-agent-hsd9r\" (UID: \"02af621e-6be7-4290-982f-8b6ee14bfd8c\") " pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092617    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1395e458-dd6f-4c0c-9fdd-bd31989597de-multus-cni-dir\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.092631    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret podName:e2017749-1a67-4077-8ae7-126c4fa84cf2 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:20.592615208 +0000 UTC m=+2.966939012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret") pod "global-pull-secret-syncer-mgvxv" (UID: "e2017749-1a67-4077-8ae7-126c4fa84cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:20.104654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.092964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03550965-bdc0-4bfe-8915-b27d8e65b0d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.093244    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b736c890-2a7a-4bfa-8336-0add1879d1bb-tmp\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.093494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02af621e-6be7-4290-982f-8b6ee14bfd8c-agent-certs\") pod \"konnectivity-agent-hsd9r\" (UID: \"02af621e-6be7-4290-982f-8b6ee14bfd8c\") " pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.095473    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b736c890-2a7a-4bfa-8336-0add1879d1bb-etc-tuned\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.097507    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrxb\" (UniqueName: \"kubernetes.io/projected/1395e458-dd6f-4c0c-9fdd-bd31989597de-kube-api-access-zbrxb\") pod \"multus-b9qzc\" (UID: \"1395e458-dd6f-4c0c-9fdd-bd31989597de\") " pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.098358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzs78\" (UniqueName: \"kubernetes.io/projected/03550965-bdc0-4bfe-8915-b27d8e65b0d8-kube-api-access-mzs78\") pod \"multus-additional-cni-plugins-t4dct\" (UID: \"03550965-bdc0-4bfe-8915-b27d8e65b0d8\") " pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.098365    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfj6\" (UniqueName: \"kubernetes.io/projected/b4059f83-eb9e-4c96-b934-0fb98b2716da-kube-api-access-wrfj6\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.098589    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkdj\" (UniqueName: \"kubernetes.io/projected/b875a80b-0dd4-4af6-8222-2f86c18b50a8-kube-api-access-5gkdj\") pod \"node-resolver-rnh76\" (UID: \"b875a80b-0dd4-4af6-8222-2f86c18b50a8\") " pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.100996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7jf\" (UniqueName: \"kubernetes.io/projected/257e50ad-c41f-420c-b697-a1ea0d80bb2d-kube-api-access-bc7jf\") pod \"ovnkube-node-w5txw\" (UID: \"257e50ad-c41f-420c-b697-a1ea0d80bb2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.105509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.101410    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfbj\" (UniqueName: \"kubernetes.io/projected/b736c890-2a7a-4bfa-8336-0add1879d1bb-kube-api-access-dmfbj\") pod \"tuned-vv7b4\" (UID: \"b736c890-2a7a-4bfa-8336-0add1879d1bb\") " pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.126582 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.126529    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal" event={"ID":"fcde823c27ff5664b9e4fd7c2bf072bf","Type":"ContainerStarted","Data":"c83cde5926869b1acce8103758a5ad6577d0b62593570e0f4c139afac05d2d6a"}
Sep 09 21:33:20.127894 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.127867    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal" event={"ID":"2fae258013d512334aa1ba0a14e3b988","Type":"ContainerStarted","Data":"0872aac5b9df57267be97ef8d96578ad582df115c447f78a664e61171aa0a67d"}
Sep 09 21:33:20.191450 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191420    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.191450 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191464    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-device-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.191665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191492    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2708ff9c-c567-436c-a587-54c9ec3213b7-host-slash\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.191665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191523    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8651d356-a086-46cb-9f76-940a0cee764a-serviceca\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.191665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191555    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-registration-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.191665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191591    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8651d356-a086-46cb-9f76-940a0cee764a-host\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.191665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191617    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk44m\" (UniqueName: \"kubernetes.io/projected/8651d356-a086-46cb-9f76-940a0cee764a-kube-api-access-nk44m\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.191665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191651    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:20.191963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191684    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqhw2\" (UniqueName: \"kubernetes.io/projected/2708ff9c-c567-436c-a587-54c9ec3213b7-kube-api-access-vqhw2\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.191963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191714    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.191963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191755    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2708ff9c-c567-436c-a587-54c9ec3213b7-iptables-alerter-script\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.191963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-sys-fs\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.191963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191825    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-socket-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.191963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.191859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2kz\" (UniqueName: \"kubernetes.io/projected/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-kube-api-access-pn2kz\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.192493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192187    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.192493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192268    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-device-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.192493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192274    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2708ff9c-c567-436c-a587-54c9ec3213b7-host-slash\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.192493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192303    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8651d356-a086-46cb-9f76-940a0cee764a-host\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.192493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192324    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-socket-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.192493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192318    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-registration-dir\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.192493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-sys-fs\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.192798 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192504    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.192798 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192633    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8651d356-a086-46cb-9f76-940a0cee764a-serviceca\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.192798 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.192732    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2708ff9c-c567-436c-a587-54c9ec3213b7-iptables-alerter-script\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.197988 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.197966    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:20.198117 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.197995    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:20.198117 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.198008    2585 projected.go:196] Error preparing data for projected volume kube-api-access-hc6lp for pod openshift-network-diagnostics/network-check-target-g6bm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:20.198117 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.198077    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp podName:7a1c2527-af7a-4e75-8e48-530b98632a9e nodeName:}" failed. No retries permitted until 2026-09-09 21:33:20.698062016 +0000 UTC m=+3.072385819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hc6lp" (UniqueName: "kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp") pod "network-check-target-g6bm5" (UID: "7a1c2527-af7a-4e75-8e48-530b98632a9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:20.201181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.201118    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqhw2\" (UniqueName: \"kubernetes.io/projected/2708ff9c-c567-436c-a587-54c9ec3213b7-kube-api-access-vqhw2\") pod \"iptables-alerter-t2qnc\" (UID: \"2708ff9c-c567-436c-a587-54c9ec3213b7\") " pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.201181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.201154    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk44m\" (UniqueName: \"kubernetes.io/projected/8651d356-a086-46cb-9f76-940a0cee764a-kube-api-access-nk44m\") pod \"node-ca-k5kh5\" (UID: \"8651d356-a086-46cb-9f76-940a0cee764a\") " pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.201362 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.201310    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2kz\" (UniqueName: \"kubernetes.io/projected/1a12f32f-ddbd-4b60-a423-7bcf3d21c19d-kube-api-access-pn2kz\") pod \"aws-ebs-csi-driver-node-d2ds8\" (UID: \"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.224951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.224924    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:20.273719 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.273690    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:20.284685 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.284655    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:20.294661 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.294639    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4dct"
Sep 09 21:33:20.300420 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.300394    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b9qzc"
Sep 09 21:33:20.309139 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.309118    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rnh76"
Sep 09 21:33:20.316917 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.316889    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vv7b4"
Sep 09 21:33:20.324574 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.324556    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8"
Sep 09 21:33:20.332234 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.332191    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k5kh5"
Sep 09 21:33:20.342039 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.342015    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t2qnc"
Sep 09 21:33:20.345419 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.345395    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:20.595216 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.594934    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:20.595216 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.594988    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:20.595216 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.595063    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:20.595216 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.595091    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:20.595216 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.595136    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs podName:b4059f83-eb9e-4c96-b934-0fb98b2716da nodeName:}" failed. No retries permitted until 2026-09-09 21:33:21.595115577 +0000 UTC m=+3.969439394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs") pod "network-metrics-daemon-qvhdr" (UID: "b4059f83-eb9e-4c96-b934-0fb98b2716da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:20.595216 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.595155    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret podName:e2017749-1a67-4077-8ae7-126c4fa84cf2 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:21.595145351 +0000 UTC m=+3.969469168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret") pod "global-pull-secret-syncer-mgvxv" (UID: "e2017749-1a67-4077-8ae7-126c4fa84cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:20.796673 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:20.796639    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:20.796855 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.796810    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:20.796855 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.796830    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:20.796855 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.796838    2585 projected.go:196] Error preparing data for projected volume kube-api-access-hc6lp for pod openshift-network-diagnostics/network-check-target-g6bm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:20.796981 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:20.796892    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp podName:7a1c2527-af7a-4e75-8e48-530b98632a9e nodeName:}" failed. No retries permitted until 2026-09-09 21:33:21.796875325 +0000 UTC m=+4.171199130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc6lp" (UniqueName: "kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp") pod "network-check-target-g6bm5" (UID: "7a1c2527-af7a-4e75-8e48-530b98632a9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:21.018839 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:21.018760    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 21:28:20 +0000 UTC" deadline="2028-04-19 16:14:22.62238419 +0000 UTC"
Sep 09 21:33:21.018839 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:21.018791    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14106h41m1.603596499s"
Sep 09 21:33:21.177557 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.177528    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb736c890_2a7a_4bfa_8336_0add1879d1bb.slice/crio-2edb57a70537a66af6f3502d7fb8ad843ad0a67cc9b7f4f5b6ad179acda9d581 WatchSource:0}: Error finding container 2edb57a70537a66af6f3502d7fb8ad843ad0a67cc9b7f4f5b6ad179acda9d581: Status 404 returned error can't find the container with id 2edb57a70537a66af6f3502d7fb8ad843ad0a67cc9b7f4f5b6ad179acda9d581
Sep 09 21:33:21.193347 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.193320    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03550965_bdc0_4bfe_8915_b27d8e65b0d8.slice/crio-6b2369de17e38627dcb1332d12b2fe4c1ba00506c4a54b08345e51fdeff540e8 WatchSource:0}: Error finding container 6b2369de17e38627dcb1332d12b2fe4c1ba00506c4a54b08345e51fdeff540e8: Status 404 returned error can't find the container with id 6b2369de17e38627dcb1332d12b2fe4c1ba00506c4a54b08345e51fdeff540e8
Sep 09 21:33:21.193924 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.193885    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2708ff9c_c567_436c_a587_54c9ec3213b7.slice/crio-0906ff6dcea956bc629e91ab702a30c3bd2de195a8d8152bb06b038b16bae5b4 WatchSource:0}: Error finding container 0906ff6dcea956bc629e91ab702a30c3bd2de195a8d8152bb06b038b16bae5b4: Status 404 returned error can't find the container with id 0906ff6dcea956bc629e91ab702a30c3bd2de195a8d8152bb06b038b16bae5b4
Sep 09 21:33:21.194842 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.194818    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257e50ad_c41f_420c_b697_a1ea0d80bb2d.slice/crio-ce6381a3260c686cff89145b8d0958b90d127fe7ade7e3f873e1767662b04603 WatchSource:0}: Error finding container ce6381a3260c686cff89145b8d0958b90d127fe7ade7e3f873e1767662b04603: Status 404 returned error can't find the container with id ce6381a3260c686cff89145b8d0958b90d127fe7ade7e3f873e1767662b04603
Sep 09 21:33:21.196926 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.196901    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8651d356_a086_46cb_9f76_940a0cee764a.slice/crio-ca6a1ca4afde22057e58acefdc48b57383c8e665289ce76b0c0b9cf81ae8e45e WatchSource:0}: Error finding container ca6a1ca4afde22057e58acefdc48b57383c8e665289ce76b0c0b9cf81ae8e45e: Status 404 returned error can't find the container with id ca6a1ca4afde22057e58acefdc48b57383c8e665289ce76b0c0b9cf81ae8e45e
Sep 09 21:33:21.198702 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.198640    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1395e458_dd6f_4c0c_9fdd_bd31989597de.slice/crio-ddc963edc0c48815b3f100e83742b3c7745ab6f337bc3fecc1c2972cadc426c9 WatchSource:0}: Error finding container ddc963edc0c48815b3f100e83742b3c7745ab6f337bc3fecc1c2972cadc426c9: Status 404 returned error can't find the container with id ddc963edc0c48815b3f100e83742b3c7745ab6f337bc3fecc1c2972cadc426c9
Sep 09 21:33:21.200379 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.200344    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02af621e_6be7_4290_982f_8b6ee14bfd8c.slice/crio-e73c6cd858f519c631c8f712449fc15a3694e454a135f691249b16be75b8c59e WatchSource:0}: Error finding container e73c6cd858f519c631c8f712449fc15a3694e454a135f691249b16be75b8c59e: Status 404 returned error can't find the container with id e73c6cd858f519c631c8f712449fc15a3694e454a135f691249b16be75b8c59e
Sep 09 21:33:21.202096 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.201913    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb875a80b_0dd4_4af6_8222_2f86c18b50a8.slice/crio-83a5439e21ba11e876b5d39f74a70e333d88c5965ed21ba7a252fc34c307de22 WatchSource:0}: Error finding container 83a5439e21ba11e876b5d39f74a70e333d88c5965ed21ba7a252fc34c307de22: Status 404 returned error can't find the container with id 83a5439e21ba11e876b5d39f74a70e333d88c5965ed21ba7a252fc34c307de22
Sep 09 21:33:21.202543 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:21.202522    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a12f32f_ddbd_4b60_a423_7bcf3d21c19d.slice/crio-64e914ce98295c0addd01371a6f0884dbca82e8dc306d90e5f97beccd9f8400e WatchSource:0}: Error finding container 64e914ce98295c0addd01371a6f0884dbca82e8dc306d90e5f97beccd9f8400e: Status 404 returned error can't find the container with id 64e914ce98295c0addd01371a6f0884dbca82e8dc306d90e5f97beccd9f8400e
Sep 09 21:33:21.602908 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:21.602641    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:21.602908 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:21.602877    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:21.603111 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:21.602798    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:21.603111 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:21.602993    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:21.603111 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:21.603005    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs podName:b4059f83-eb9e-4c96-b934-0fb98b2716da nodeName:}" failed. No retries permitted until 2026-09-09 21:33:23.602985471 +0000 UTC m=+5.977309288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs") pod "network-metrics-daemon-qvhdr" (UID: "b4059f83-eb9e-4c96-b934-0fb98b2716da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:21.603111 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:21.603027    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret podName:e2017749-1a67-4077-8ae7-126c4fa84cf2 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:23.603017611 +0000 UTC m=+5.977341413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret") pod "global-pull-secret-syncer-mgvxv" (UID: "e2017749-1a67-4077-8ae7-126c4fa84cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:21.803746 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:21.803560    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:21.803932 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:21.803768    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:21.803932 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:21.803794    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:21.803932 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:21.803808    2585 projected.go:196] Error preparing data for projected volume kube-api-access-hc6lp for pod openshift-network-diagnostics/network-check-target-g6bm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:21.803932 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:21.803866    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp podName:7a1c2527-af7a-4e75-8e48-530b98632a9e nodeName:}" failed. No retries permitted until 2026-09-09 21:33:23.803848513 +0000 UTC m=+6.178172335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc6lp" (UniqueName: "kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp") pod "network-check-target-g6bm5" (UID: "7a1c2527-af7a-4e75-8e48-530b98632a9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:22.019809 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.019711    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 21:28:20 +0000 UTC" deadline="2028-04-12 17:15:22.398489773 +0000 UTC"
Sep 09 21:33:22.019809 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.019767    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13939h42m0.378726775s"
Sep 09 21:33:22.123090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.122331    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:22.123090 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:22.122471    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:22.123090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.122981    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:22.123090 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:22.123048    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:22.123539 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.123404    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:22.123539 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:22.123464    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:22.150416 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.150319    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vv7b4" event={"ID":"b736c890-2a7a-4bfa-8336-0add1879d1bb","Type":"ContainerStarted","Data":"2edb57a70537a66af6f3502d7fb8ad843ad0a67cc9b7f4f5b6ad179acda9d581"}
Sep 09 21:33:22.168630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.168004    2585 generic.go:356] "Generic (PLEG): container finished" podID="2fae258013d512334aa1ba0a14e3b988" containerID="13826c0b684be1abb40b7f08ca9fcde7e7bf55a6c1e33a5996badb0549ff80ad" exitCode=0
Sep 09 21:33:22.168630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.168106    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal" event={"ID":"2fae258013d512334aa1ba0a14e3b988","Type":"ContainerDied","Data":"13826c0b684be1abb40b7f08ca9fcde7e7bf55a6c1e33a5996badb0549ff80ad"}
Sep 09 21:33:22.186128 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.186094    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rnh76" event={"ID":"b875a80b-0dd4-4af6-8222-2f86c18b50a8","Type":"ContainerStarted","Data":"83a5439e21ba11e876b5d39f74a70e333d88c5965ed21ba7a252fc34c307de22"}
Sep 09 21:33:22.202838 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.202782    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hsd9r" event={"ID":"02af621e-6be7-4290-982f-8b6ee14bfd8c","Type":"ContainerStarted","Data":"e73c6cd858f519c631c8f712449fc15a3694e454a135f691249b16be75b8c59e"}
Sep 09 21:33:22.210985 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.210950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b9qzc" event={"ID":"1395e458-dd6f-4c0c-9fdd-bd31989597de","Type":"ContainerStarted","Data":"ddc963edc0c48815b3f100e83742b3c7745ab6f337bc3fecc1c2972cadc426c9"}
Sep 09 21:33:22.215220 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.215171    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k5kh5" event={"ID":"8651d356-a086-46cb-9f76-940a0cee764a","Type":"ContainerStarted","Data":"ca6a1ca4afde22057e58acefdc48b57383c8e665289ce76b0c0b9cf81ae8e45e"}
Sep 09 21:33:22.219271 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.219240    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t2qnc" event={"ID":"2708ff9c-c567-436c-a587-54c9ec3213b7","Type":"ContainerStarted","Data":"0906ff6dcea956bc629e91ab702a30c3bd2de195a8d8152bb06b038b16bae5b4"}
Sep 09 21:33:22.240881 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.240847    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal" event={"ID":"fcde823c27ff5664b9e4fd7c2bf072bf","Type":"ContainerStarted","Data":"a6f6e9ed54e21f6fe4674f7c16ac50625f01d05a827e1edbb3c29b679a1dbdb4"}
Sep 09 21:33:22.255108 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.255040    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8" event={"ID":"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d","Type":"ContainerStarted","Data":"64e914ce98295c0addd01371a6f0884dbca82e8dc306d90e5f97beccd9f8400e"}
Sep 09 21:33:22.266976 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.266865    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"ce6381a3260c686cff89145b8d0958b90d127fe7ade7e3f873e1767662b04603"}
Sep 09 21:33:22.273066 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:22.273035    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerStarted","Data":"6b2369de17e38627dcb1332d12b2fe4c1ba00506c4a54b08345e51fdeff540e8"}
Sep 09 21:33:23.296393 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:23.296354    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal" event={"ID":"2fae258013d512334aa1ba0a14e3b988","Type":"ContainerStarted","Data":"544d49966d6eb41eae0bc2d2233004bbe25167e36accf32a98b858f06cd003e5"}
Sep 09 21:33:23.311325 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:23.311267    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal" podStartSLOduration=4.311246536 podStartE2EDuration="4.311246536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:22.254506453 +0000 UTC m=+4.628830279" watchObservedRunningTime="2026-09-09 21:33:23.311246536 +0000 UTC m=+5.685570363"
Sep 09 21:33:23.311493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:23.311434    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal" podStartSLOduration=4.311426658 podStartE2EDuration="4.311426658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:23.31061198 +0000 UTC m=+5.684935816" watchObservedRunningTime="2026-09-09 21:33:23.311426658 +0000 UTC m=+5.685750482"
Sep 09 21:33:23.619567 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:23.619482    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:23.619567 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:23.619532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:23.619792 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:23.619644    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:23.619792 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:23.619704    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret podName:e2017749-1a67-4077-8ae7-126c4fa84cf2 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:27.61968369 +0000 UTC m=+9.994007495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret") pod "global-pull-secret-syncer-mgvxv" (UID: "e2017749-1a67-4077-8ae7-126c4fa84cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:23.620099 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:23.620078    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:23.620185 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:23.620131    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs podName:b4059f83-eb9e-4c96-b934-0fb98b2716da nodeName:}" failed. No retries permitted until 2026-09-09 21:33:27.620118189 +0000 UTC m=+9.994441997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs") pod "network-metrics-daemon-qvhdr" (UID: "b4059f83-eb9e-4c96-b934-0fb98b2716da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:23.821176 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:23.820989    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:23.821176 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:23.821154    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:23.821176 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:23.821173    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:23.821176 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:23.821186    2585 projected.go:196] Error preparing data for projected volume kube-api-access-hc6lp for pod openshift-network-diagnostics/network-check-target-g6bm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:23.821523 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:23.821260    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp podName:7a1c2527-af7a-4e75-8e48-530b98632a9e nodeName:}" failed. No retries permitted until 2026-09-09 21:33:27.821241935 +0000 UTC m=+10.195565740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc6lp" (UniqueName: "kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp") pod "network-check-target-g6bm5" (UID: "7a1c2527-af7a-4e75-8e48-530b98632a9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:24.123351 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:24.123315    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:24.123529 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:24.123445    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:24.123845 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:24.123826    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:24.123955 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:24.123934    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:24.124317 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:24.124302    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:24.124438 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:24.124410    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:26.124677 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:26.124641    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:26.125138 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:26.124787    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:26.125220 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:26.125176    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:26.125330 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:26.125293    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:26.125429 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:26.125358    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:26.125429 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:26.125421    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:27.656380 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:27.656333    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:27.656860 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:27.656396    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:27.656860 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:27.656530    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:27.656860 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:27.656580    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret podName:e2017749-1a67-4077-8ae7-126c4fa84cf2 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:35.656568132 +0000 UTC m=+18.030891947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret") pod "global-pull-secret-syncer-mgvxv" (UID: "e2017749-1a67-4077-8ae7-126c4fa84cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:27.656989 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:27.656945    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:27.657022 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:27.656996    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs podName:b4059f83-eb9e-4c96-b934-0fb98b2716da nodeName:}" failed. No retries permitted until 2026-09-09 21:33:35.656980156 +0000 UTC m=+18.031303975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs") pod "network-metrics-daemon-qvhdr" (UID: "b4059f83-eb9e-4c96-b934-0fb98b2716da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:27.857295 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:27.857256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:27.857486 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:27.857448    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:27.857486 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:27.857466    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:27.857486 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:27.857478    2585 projected.go:196] Error preparing data for projected volume kube-api-access-hc6lp for pod openshift-network-diagnostics/network-check-target-g6bm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:27.857653 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:27.857541    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp podName:7a1c2527-af7a-4e75-8e48-530b98632a9e nodeName:}" failed. No retries permitted until 2026-09-09 21:33:35.857524363 +0000 UTC m=+18.231848180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc6lp" (UniqueName: "kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp") pod "network-check-target-g6bm5" (UID: "7a1c2527-af7a-4e75-8e48-530b98632a9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:28.123063 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:28.122984    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:28.123293 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:28.123105    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:28.123293 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:28.123154    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:28.123293 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:28.123235    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:28.123588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:28.123573    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:28.123674 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:28.123657    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:30.124134 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:30.124095    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:30.124612 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:30.124244    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:30.124612 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:30.124608    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:30.124722 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:30.124703    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:30.124772 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:30.124765    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:30.124846 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:30.124830    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:32.121866 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:32.121833    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:32.122303 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:32.121834    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:32.122303 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:32.121967    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:32.122303 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:32.122007    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:32.122303 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:32.122102    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:32.122303 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:32.122175    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:34.122186 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:34.122116    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:34.122645 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:34.122271    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:34.122708 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:34.122656    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:34.122708 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:34.122689    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:34.122807 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:34.122775    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:34.122897 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:34.122868    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:35.715845 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:35.715803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:35.716269 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:35.715856    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:35.716269 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:35.715963    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:35.716269 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:35.715964    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:35.716269 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:35.716027    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs podName:b4059f83-eb9e-4c96-b934-0fb98b2716da nodeName:}" failed. No retries permitted until 2026-09-09 21:33:51.716009396 +0000 UTC m=+34.090333202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs") pod "network-metrics-daemon-qvhdr" (UID: "b4059f83-eb9e-4c96-b934-0fb98b2716da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:35.716269 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:35.716041    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret podName:e2017749-1a67-4077-8ae7-126c4fa84cf2 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:51.716035015 +0000 UTC m=+34.090358818 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret") pod "global-pull-secret-syncer-mgvxv" (UID: "e2017749-1a67-4077-8ae7-126c4fa84cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:35.917878 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:35.917836    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:35.918028 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:35.917973    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:35.918028 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:35.917994    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:35.918028 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:35.918007    2585 projected.go:196] Error preparing data for projected volume kube-api-access-hc6lp for pod openshift-network-diagnostics/network-check-target-g6bm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:35.918157 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:35.918067    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp podName:7a1c2527-af7a-4e75-8e48-530b98632a9e nodeName:}" failed. No retries permitted until 2026-09-09 21:33:51.91804756 +0000 UTC m=+34.292371367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc6lp" (UniqueName: "kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp") pod "network-check-target-g6bm5" (UID: "7a1c2527-af7a-4e75-8e48-530b98632a9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:36.122056 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:36.121966    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:36.122225 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:36.121966    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:36.122225 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:36.122116    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:36.122225 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:36.121974    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:36.122225 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:36.122190    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:36.122426 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:36.122280    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:38.122259 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:38.122199    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:38.122650 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:38.122317    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:38.122650 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:38.122335    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:38.122650 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:38.122398    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:38.122650 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:38.122442    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:38.122650 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:38.122568    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:39.333455 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.333428    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b9qzc" event={"ID":"1395e458-dd6f-4c0c-9fdd-bd31989597de","Type":"ContainerStarted","Data":"4c8494e602c6b3a00609fd9a816e5948a788ccf43cb94ea639d75ce84c3ad4b5"}
Sep 09 21:33:39.335056 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.335032    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k5kh5" event={"ID":"8651d356-a086-46cb-9f76-940a0cee764a","Type":"ContainerStarted","Data":"8e9419f65cdfa0e607e17b019e6aa79dc07abaa963676833bd5f5a53eda53489"}
Sep 09 21:33:39.336902 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.336871    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8" event={"ID":"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d","Type":"ContainerStarted","Data":"3697828ed90fd8fda6b913f5946e051b3d5c7adfc657512bce561d74490f8424"}
Sep 09 21:33:39.341260 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.341239    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"b0c31841babf1afbb68e9a665918a695b7eec55eae840bff421e8de842526e20"}
Sep 09 21:33:39.342713 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.342690    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerStarted","Data":"87875c239aa13a6e51acd3a796a29a10b39dc0963dbfd2926992855c0710660f"}
Sep 09 21:33:39.344431 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.344408    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vv7b4" event={"ID":"b736c890-2a7a-4bfa-8336-0add1879d1bb","Type":"ContainerStarted","Data":"07bcb564f2e491ffa806d34eb9c5b70198cf6b295fc3f0279aa047c1030f30f3"}
Sep 09 21:33:39.345964 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.345940    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rnh76" event={"ID":"b875a80b-0dd4-4af6-8222-2f86c18b50a8","Type":"ContainerStarted","Data":"12e177e30a001a987a2761261d8f83443e42263ced116f0dab55a0a3fcb1c498"}
Sep 09 21:33:39.347553 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.347535    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hsd9r" event={"ID":"02af621e-6be7-4290-982f-8b6ee14bfd8c","Type":"ContainerStarted","Data":"526610482ed73e55d09b5354208a2e1fd9e671a1c3029abe486ad67c9d503332"}
Sep 09 21:33:39.364041 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.363985    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b9qzc" podStartSLOduration=3.630770777 podStartE2EDuration="21.363965792s" totalImagesPullingTime="17.733195015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:39.349298429 +0000 UTC m=+21.723622257" watchObservedRunningTime="2026-09-09 21:33:39.363965792 +0000 UTC m=+21.738289618"
Sep 09 21:33:39.364381 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.364344    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vv7b4" podStartSLOduration=3.627922916 podStartE2EDuration="21.364328727s" totalImagesPullingTime="17.736405811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:39.363680952 +0000 UTC m=+21.738004776" watchObservedRunningTime="2026-09-09 21:33:39.364328727 +0000 UTC m=+21.738652548"
Sep 09 21:33:39.375580 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.375544    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hsd9r" podStartSLOduration=11.152987395 podStartE2EDuration="21.375531136s" totalImagesPullingTime="10.222543741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:39.375300365 +0000 UTC m=+21.749624206" watchObservedRunningTime="2026-09-09 21:33:39.375531136 +0000 UTC m=+21.749854959"
Sep 09 21:33:39.389620 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.389579    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rnh76" podStartSLOduration=3.671084035 podStartE2EDuration="21.389565696s" totalImagesPullingTime="17.718481661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:39.389500009 +0000 UTC m=+21.763823833" watchObservedRunningTime="2026-09-09 21:33:39.389565696 +0000 UTC m=+21.763889520"
Sep 09 21:33:39.403629 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.403588    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k5kh5" podStartSLOduration=3.706407322 podStartE2EDuration="21.403573153s" totalImagesPullingTime="17.697165831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:39.403140747 +0000 UTC m=+21.777464571" watchObservedRunningTime="2026-09-09 21:33:39.403573153 +0000 UTC m=+21.777897000"
Sep 09 21:33:39.474244 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.474155    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:39.547361 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:39.547145    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:40.121604 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.121530    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:40.121604 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.121550    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:40.121831 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:40.121643    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:40.121831 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:40.121694    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:40.121831 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.121741    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:40.121831 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:40.121805    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:40.353728 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.353699    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:33:40.354489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.354021    2585 generic.go:356] "Generic (PLEG): container finished" podID="257e50ad-c41f-420c-b697-a1ea0d80bb2d" containerID="d8adf916b8643aeb895114095c0338d3576e4ebb955f41a35414b7008d124669" exitCode=1
Sep 09 21:33:40.354489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.354086    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"84026db839b0e9c0717b6189b882b3c5b62846437fb73c4c206536c6d570c8af"}
Sep 09 21:33:40.354489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.354149    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"abfe336ca047d79786eec02ab1d1579862123a0d2d025b52981f573648489682"}
Sep 09 21:33:40.354489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.354233    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"0f6809b0b08432a175d9b259acb244ab9a1c9330ba24ddd506635f9929da8a45"}
Sep 09 21:33:40.354489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.354273    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"2459e1d8f7f1fe4a6c1c39233044b9bf2092bdbfd8f5f184890bbd0447510483"}
Sep 09 21:33:40.354489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.354301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerDied","Data":"d8adf916b8643aeb895114095c0338d3576e4ebb955f41a35414b7008d124669"}
Sep 09 21:33:40.355539 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.355516    2585 generic.go:356] "Generic (PLEG): container finished" podID="03550965-bdc0-4bfe-8915-b27d8e65b0d8" containerID="87875c239aa13a6e51acd3a796a29a10b39dc0963dbfd2926992855c0710660f" exitCode=0
Sep 09 21:33:40.355633 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.355619    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerDied","Data":"87875c239aa13a6e51acd3a796a29a10b39dc0963dbfd2926992855c0710660f"}
Sep 09 21:33:40.518043 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:40.518019    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 21:33:41.053216 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:41.053131    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T21:33:40.518039864Z","UUID":"28217a4a-5b4c-4222-aa31-ee446feca87f","Handler":null,"Name":"","Endpoint":""}
Sep 09 21:33:41.054483 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:41.054467    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 21:33:41.054568 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:41.054493    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 21:33:41.360761 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:41.360678    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t2qnc" event={"ID":"2708ff9c-c567-436c-a587-54c9ec3213b7","Type":"ContainerStarted","Data":"fea004a8ee4bf24871d04571475eab762804e6dea94c5fe443e81733b0265978"}
Sep 09 21:33:41.363679 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:41.363614    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8" event={"ID":"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d","Type":"ContainerStarted","Data":"b10afb0577a759d30af6f654c0db92857e76d65cea9a4adeeedd1cee3aad7ec4"}
Sep 09 21:33:41.374938 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:41.374893    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t2qnc" podStartSLOduration=5.645427777 podStartE2EDuration="23.374878041s" totalImagesPullingTime="17.729450264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:41.374630755 +0000 UTC m=+23.748954581" watchObservedRunningTime="2026-09-09 21:33:41.374878041 +0000 UTC m=+23.749201865"
Sep 09 21:33:41.594427 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:41.594389    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:41.651582 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:41.651502    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:42.122360 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.122332    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:42.122360 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.122358    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:42.123382 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.122332    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:42.123382 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:42.122490    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:42.123382 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:42.122643    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:42.123382 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:42.122737    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:42.336986 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.336947    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:42.368100 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.368061    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8" event={"ID":"1a12f32f-ddbd-4b60-a423-7bcf3d21c19d","Type":"ContainerStarted","Data":"4e4b358e94451643a696c9347d232de2d2a10430ba9086fffdac21e2f8f04478"}
Sep 09 21:33:42.372012 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.371853    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:33:42.372346 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.372322    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"f4591943ef0dc6f34c618d294edd899aadd31f0bab9f8b5407c97733da0c845e"}
Sep 09 21:33:42.385267 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.385200    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2ds8" podStartSLOduration=3.891304038 podStartE2EDuration="24.385183502s" totalImagesPullingTime="20.493879464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:42.38456514 +0000 UTC m=+24.758888965" watchObservedRunningTime="2026-09-09 21:33:42.385183502 +0000 UTC m=+24.759507326"
Sep 09 21:33:42.398688 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:42.398650    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:43.110484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:43.110447    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:43.111270 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:43.111244    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:43.375478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:43.375396    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:43.375960 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:43.375937    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hsd9r"
Sep 09 21:33:43.439450 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:43.439421    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:43.925485 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:43.925301    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:43.974181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:43.974143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:44.121565 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:44.121529    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:44.121565 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:44.121548    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:44.121787 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:44.121529    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:44.121787 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:44.121654    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:44.121787 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:44.121727    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:44.121927 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:44.121818    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:45.383389 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:45.383304    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:33:45.384107 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:45.383731    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"57344c01a5fdef473faba2e8ebc87036f8d209e74e44ca2881f9f27de6d9ae9a"}
Sep 09 21:33:45.384107 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:45.383981    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:45.384233 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:45.384183    2585 scope.go:117] "RemoveContainer" containerID="d8adf916b8643aeb895114095c0338d3576e4ebb955f41a35414b7008d124669"
Sep 09 21:33:45.385951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:45.385912    2585 generic.go:356] "Generic (PLEG): container finished" podID="03550965-bdc0-4bfe-8915-b27d8e65b0d8" containerID="dc5cfc30e0f34ffde74f532371e49cfebc3cf9f387e3d6ab364cae745eabf10e" exitCode=0
Sep 09 21:33:45.386054 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:45.385983    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerDied","Data":"dc5cfc30e0f34ffde74f532371e49cfebc3cf9f387e3d6ab364cae745eabf10e"}
Sep 09 21:33:45.400372 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:45.400349    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:46.121557 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.121523    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:46.121723 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.121522    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:46.121723 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:46.121680    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:46.121821 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:46.121780    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:46.121878 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.121865    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:46.121958 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:46.121941    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:46.396374 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.396159    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:33:46.396808 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.396773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" event={"ID":"257e50ad-c41f-420c-b697-a1ea0d80bb2d","Type":"ContainerStarted","Data":"0694abe12c9c8bc1b7daaff1ef8df241863dd763f2b2d074ca2365eed5f10b23"}
Sep 09 21:33:46.397366 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.397093    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:46.397366 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.397184    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:46.418998 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.418964    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:33:46.422852 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.422434    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw" podStartSLOduration=10.655917022 podStartE2EDuration="28.422418949s" totalImagesPullingTime="17.766501927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:46.420835863 +0000 UTC m=+28.795159692" watchObservedRunningTime="2026-09-09 21:33:46.422418949 +0000 UTC m=+28.796742795"
Sep 09 21:33:46.597915 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.597882    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mgvxv"]
Sep 09 21:33:46.598080 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.598007    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:46.598153 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:46.598133    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:46.600853 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.600829    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvhdr"]
Sep 09 21:33:46.601028 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.600920    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:46.601028 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:46.601012    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:46.601475 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.601455    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6bm5"]
Sep 09 21:33:46.601581 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:46.601541    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:46.601661 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:46.601641    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:47.401262 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:47.401229    2585 generic.go:356] "Generic (PLEG): container finished" podID="03550965-bdc0-4bfe-8915-b27d8e65b0d8" containerID="9c0c79da9cc2e01db2a018eaa808a2f2367674bfddb498a7d38afdd9b6bf9bb2" exitCode=0
Sep 09 21:33:47.401644 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:47.401327    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerDied","Data":"9c0c79da9cc2e01db2a018eaa808a2f2367674bfddb498a7d38afdd9b6bf9bb2"}
Sep 09 21:33:48.122890 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:48.122853    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:48.123074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:48.122900    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:48.123074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:48.122967    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:48.123074 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:48.122996    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:48.123074 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:48.123045    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:48.123303 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:48.123102    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:49.409164 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:49.409132    2585 generic.go:356] "Generic (PLEG): container finished" podID="03550965-bdc0-4bfe-8915-b27d8e65b0d8" containerID="0b385c7a135104b051deea72c189d5017f43a7d9051a15d50d6d9961471fdb7f" exitCode=0
Sep 09 21:33:49.409647 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:49.409177    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerDied","Data":"0b385c7a135104b051deea72c189d5017f43a7d9051a15d50d6d9961471fdb7f"}
Sep 09 21:33:49.463976 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:49.463942    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:49.781456 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:49.781359    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:50.122078 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:50.122003    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:50.122078 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:50.122044    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:50.122324 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:50.122084    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:50.122324 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:50.122227    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgvxv" podUID="e2017749-1a67-4077-8ae7-126c4fa84cf2"
Sep 09 21:33:50.122324 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:50.122308    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6bm5" podUID="7a1c2527-af7a-4e75-8e48-530b98632a9e"
Sep 09 21:33:50.122481 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:50.122415    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvhdr" podUID="b4059f83-eb9e-4c96-b934-0fb98b2716da"
Sep 09 21:33:51.727693 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.727663    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:51.728116 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.727703    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:51.728116 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:51.727815    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:51.728116 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:51.727813    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:51.728116 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:51.727870    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret podName:e2017749-1a67-4077-8ae7-126c4fa84cf2 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:23.727852089 +0000 UTC m=+66.102175908 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret") pod "global-pull-secret-syncer-mgvxv" (UID: "e2017749-1a67-4077-8ae7-126c4fa84cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:51.728116 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:51.727890    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs podName:b4059f83-eb9e-4c96-b934-0fb98b2716da nodeName:}" failed. No retries permitted until 2026-09-09 21:34:23.727879885 +0000 UTC m=+66.102203692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs") pod "network-metrics-daemon-qvhdr" (UID: "b4059f83-eb9e-4c96-b934-0fb98b2716da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:51.925712 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.925636    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeReady"
Sep 09 21:33:51.925835 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.925791    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 21:33:51.929833 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.929810    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:51.929972 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:51.929956    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:51.930024 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:51.929980    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:51.930024 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:51.929995    2585 projected.go:196] Error preparing data for projected volume kube-api-access-hc6lp for pod openshift-network-diagnostics/network-check-target-g6bm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:51.930103 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:51.930048    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp podName:7a1c2527-af7a-4e75-8e48-530b98632a9e nodeName:}" failed. No retries permitted until 2026-09-09 21:34:23.930034407 +0000 UTC m=+66.304358223 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc6lp" (UniqueName: "kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp") pod "network-check-target-g6bm5" (UID: "7a1c2527-af7a-4e75-8e48-530b98632a9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:51.970903 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.970873    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sdxrh"]
Sep 09 21:33:51.973362 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.973342    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l9tcv"]
Sep 09 21:33:51.973489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.973480    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:51.975143 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.975126    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l9tcv"
Sep 09 21:33:51.975825 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.975801    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 21:33:51.975946 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.975848    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 21:33:51.976014 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.975961    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4v4f\""
Sep 09 21:33:51.977645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.977287    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bhsnj\""
Sep 09 21:33:51.977645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.977294    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 21:33:51.977645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.977316    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 21:33:51.977645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.977404    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 21:33:51.980994 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.980972    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l9tcv"]
Sep 09 21:33:51.984553 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:51.984352    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sdxrh"]
Sep 09 21:33:52.073118 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.073054    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:52.122125 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.122086    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:33:52.122318 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.122097    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:33:52.122318 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.122254    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:33:52.124875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.124846    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 21:33:52.124992 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.124891    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 21:33:52.124992 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.124923    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 21:33:52.124992 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.124961    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kms6v\""
Sep 09 21:33:52.124992 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.124977    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5t55x\""
Sep 09 21:33:52.125155 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.125102    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 21:33:52.130895 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.130875    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.130993 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.130910    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqknn\" (UniqueName: \"kubernetes.io/projected/b58acc59-65cc-4030-af91-741e1938dc2d-kube-api-access-kqknn\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.130993 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.130938    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aee9510-534e-46a2-af83-db4a01b114aa-cert\") pod \"ingress-canary-l9tcv\" (UID: \"2aee9510-534e-46a2-af83-db4a01b114aa\") " pod="openshift-ingress-canary/ingress-canary-l9tcv"
Sep 09 21:33:52.131079 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.131052    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b58acc59-65cc-4030-af91-741e1938dc2d-config-volume\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.131125 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.131095    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58acc59-65cc-4030-af91-741e1938dc2d-tmp-dir\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.131166 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.131137    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgq5b\" (UniqueName: \"kubernetes.io/projected/2aee9510-534e-46a2-af83-db4a01b114aa-kube-api-access-kgq5b\") pod \"ingress-canary-l9tcv\" (UID: \"2aee9510-534e-46a2-af83-db4a01b114aa\") " pod="openshift-ingress-canary/ingress-canary-l9tcv"
Sep 09 21:33:52.231929 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.231839    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.231929 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.231894    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqknn\" (UniqueName: \"kubernetes.io/projected/b58acc59-65cc-4030-af91-741e1938dc2d-kube-api-access-kqknn\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.232135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.231962    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aee9510-534e-46a2-af83-db4a01b114aa-cert\") pod \"ingress-canary-l9tcv\" (UID: \"2aee9510-534e-46a2-af83-db4a01b114aa\") " pod="openshift-ingress-canary/ingress-canary-l9tcv"
Sep 09 21:33:52.232135 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:52.232006    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:52.232135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.232020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b58acc59-65cc-4030-af91-741e1938dc2d-config-volume\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.232135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.232054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58acc59-65cc-4030-af91-741e1938dc2d-tmp-dir\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.232135 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:52.232074    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls podName:b58acc59-65cc-4030-af91-741e1938dc2d nodeName:}" failed. No retries permitted until 2026-09-09 21:33:52.732051805 +0000 UTC m=+35.106375629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls") pod "dns-default-sdxrh" (UID: "b58acc59-65cc-4030-af91-741e1938dc2d") : secret "dns-default-metrics-tls" not found
Sep 09 21:33:52.232135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.232118    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgq5b\" (UniqueName: \"kubernetes.io/projected/2aee9510-534e-46a2-af83-db4a01b114aa-kube-api-access-kgq5b\") pod \"ingress-canary-l9tcv\" (UID: \"2aee9510-534e-46a2-af83-db4a01b114aa\") " pod="openshift-ingress-canary/ingress-canary-l9tcv"
Sep 09 21:33:52.232654 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.232631    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58acc59-65cc-4030-af91-741e1938dc2d-tmp-dir\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.232949 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.232923    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b58acc59-65cc-4030-af91-741e1938dc2d-config-volume\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.236955 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.236935    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aee9510-534e-46a2-af83-db4a01b114aa-cert\") pod \"ingress-canary-l9tcv\" (UID: \"2aee9510-534e-46a2-af83-db4a01b114aa\") " pod="openshift-ingress-canary/ingress-canary-l9tcv"
Sep 09 21:33:52.243273 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.243251    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqknn\" (UniqueName: \"kubernetes.io/projected/b58acc59-65cc-4030-af91-741e1938dc2d-kube-api-access-kqknn\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.244721 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.244690    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgq5b\" (UniqueName: \"kubernetes.io/projected/2aee9510-534e-46a2-af83-db4a01b114aa-kube-api-access-kgq5b\") pod \"ingress-canary-l9tcv\" (UID: \"2aee9510-534e-46a2-af83-db4a01b114aa\") " pod="openshift-ingress-canary/ingress-canary-l9tcv"
Sep 09 21:33:52.294626 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.294591    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l9tcv"
Sep 09 21:33:52.736907 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:52.736866    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:52.737331 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:52.737023    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:52.737331 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:52.737089    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls podName:b58acc59-65cc-4030-af91-741e1938dc2d nodeName:}" failed. No retries permitted until 2026-09-09 21:33:53.737069284 +0000 UTC m=+36.111393090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls") pod "dns-default-sdxrh" (UID: "b58acc59-65cc-4030-af91-741e1938dc2d") : secret "dns-default-metrics-tls" not found
Sep 09 21:33:53.650672 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:53.650595    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l9tcv"]
Sep 09 21:33:53.653924 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:33:53.653880    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aee9510_534e_46a2_af83_db4a01b114aa.slice/crio-35c5c2ebd11e2dac7848c6eb24e931ce325fa0a453d5a8fad816fb0e5c5f1f09 WatchSource:0}: Error finding container 35c5c2ebd11e2dac7848c6eb24e931ce325fa0a453d5a8fad816fb0e5c5f1f09: Status 404 returned error can't find the container with id 35c5c2ebd11e2dac7848c6eb24e931ce325fa0a453d5a8fad816fb0e5c5f1f09
Sep 09 21:33:53.744917 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:53.744881    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:53.745600 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:53.745045    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:53.745600 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:53.745120    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls podName:b58acc59-65cc-4030-af91-741e1938dc2d nodeName:}" failed. No retries permitted until 2026-09-09 21:33:55.745100148 +0000 UTC m=+38.119423974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls") pod "dns-default-sdxrh" (UID: "b58acc59-65cc-4030-af91-741e1938dc2d") : secret "dns-default-metrics-tls" not found
Sep 09 21:33:54.225908 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:54.225870    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:54.326373 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:54.326338    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:54.424389 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:54.424351    2585 generic.go:356] "Generic (PLEG): container finished" podID="03550965-bdc0-4bfe-8915-b27d8e65b0d8" containerID="a628e1cc8aa38bff68d68e62ba433e90924bfe832824f05a9b11b719c5077c3e" exitCode=0
Sep 09 21:33:54.424574 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:54.424436    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerDied","Data":"a628e1cc8aa38bff68d68e62ba433e90924bfe832824f05a9b11b719c5077c3e"}
Sep 09 21:33:54.425605 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:54.425580    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l9tcv" event={"ID":"2aee9510-534e-46a2-af83-db4a01b114aa","Type":"ContainerStarted","Data":"35c5c2ebd11e2dac7848c6eb24e931ce325fa0a453d5a8fad816fb0e5c5f1f09"}
Sep 09 21:33:55.432147 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:55.431908    2585 generic.go:356] "Generic (PLEG): container finished" podID="03550965-bdc0-4bfe-8915-b27d8e65b0d8" containerID="9e76f12e18c121130f9ceae696961aa4a258eae9da7b2c581d5955360066863a" exitCode=0
Sep 09 21:33:55.432147 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:55.431947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerDied","Data":"9e76f12e18c121130f9ceae696961aa4a258eae9da7b2c581d5955360066863a"}
Sep 09 21:33:55.762185 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:55.762119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:55.762365 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:55.762284    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:55.762365 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:55.762353    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls podName:b58acc59-65cc-4030-af91-741e1938dc2d nodeName:}" failed. No retries permitted until 2026-09-09 21:33:59.762335907 +0000 UTC m=+42.136659740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls") pod "dns-default-sdxrh" (UID: "b58acc59-65cc-4030-af91-741e1938dc2d") : secret "dns-default-metrics-tls" not found
Sep 09 21:33:56.435727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:56.435692    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l9tcv" event={"ID":"2aee9510-534e-46a2-af83-db4a01b114aa","Type":"ContainerStarted","Data":"1975b3cce580c3d2df9b6a445f5b0cadbe1b878de7ac3868727cdd5d9597c872"}
Sep 09 21:33:56.439200 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:56.439178    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4dct" event={"ID":"03550965-bdc0-4bfe-8915-b27d8e65b0d8","Type":"ContainerStarted","Data":"dea9692788ccd11410b21a002cc35115dee2f9a3d0e576c0a96f7f36b13dbfcf"}
Sep 09 21:33:56.451380 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:56.451337    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l9tcv" podStartSLOduration=3.090754409 podStartE2EDuration="5.451324269s" totalImagesPullingTime="2.36056986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:56.450705409 +0000 UTC m=+38.825029233" watchObservedRunningTime="2026-09-09 21:33:56.451324269 +0000 UTC m=+38.825648092"
Sep 09 21:33:56.455311 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:56.455289    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:56.469472 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:56.469436    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t4dct" podStartSLOduration=10.299550112 podStartE2EDuration="38.469414235s" totalImagesPullingTime="28.169864123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:56.468392764 +0000 UTC m=+38.842716588" watchObservedRunningTime="2026-09-09 21:33:56.469414235 +0000 UTC m=+38.843738065"
Sep 09 21:33:56.512823 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:56.512801    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:56.587289 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:56.587266    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:56.648892 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:56.648863    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:57.265955 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:57.265928    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:57.317373 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:57.317323    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:59.791081 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:33:59.791043    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:33:59.791521 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:59.791151    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:59.791521 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:33:59.791202    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls podName:b58acc59-65cc-4030-af91-741e1938dc2d nodeName:}" failed. No retries permitted until 2026-09-09 21:34:07.791188139 +0000 UTC m=+50.165511945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls") pod "dns-default-sdxrh" (UID: "b58acc59-65cc-4030-af91-741e1938dc2d") : secret "dns-default-metrics-tls" not found
Sep 09 21:34:00.303040 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:00.303005    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:00.361721 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:00.361690    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:05.611751 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.611701    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"]
Sep 09 21:34:05.654905 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.654866    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-ntfn6"]
Sep 09 21:34:05.655079 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.654982    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.657467 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.657402    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 21:34:05.657467 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.657441    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 21:34:05.657467 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.657440    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 21:34:05.657467 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.657449    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:34:05.658789 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.658290    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5rrls\""
Sep 09 21:34:05.670369 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.670350    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"]
Sep 09 21:34:05.670464 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.670379    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-ntfn6"]
Sep 09 21:34:05.670464 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.670445    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.672558 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.672540    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 21:34:05.672658 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.672593    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 21:34:05.672658 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.672629    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 21:34:05.672884 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.672869    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 21:34:05.673158 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.673144    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qtzl6\""
Sep 09 21:34:05.677527 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.677510    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 21:34:05.728832 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.728801    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-serving-cert\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.728955 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.728858    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksbj8\" (UniqueName: \"kubernetes.io/projected/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-kube-api-access-ksbj8\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.728955 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.728934    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-config\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.739427 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.739397    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:05.798576 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.798541    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:05.815801 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.815777    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"]
Sep 09 21:34:05.829468 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829443    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f17101f-b5eb-4af2-bc07-a05e07df482e-tmp\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.829577 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829475    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f17101f-b5eb-4af2-bc07-a05e07df482e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.829577 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829492    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f17101f-b5eb-4af2-bc07-a05e07df482e-serving-cert\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.829577 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829514    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbj8\" (UniqueName: \"kubernetes.io/projected/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-kube-api-access-ksbj8\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.829736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829595    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f17101f-b5eb-4af2-bc07-a05e07df482e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.829736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829634    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchst\" (UniqueName: \"kubernetes.io/projected/5f17101f-b5eb-4af2-bc07-a05e07df482e-kube-api-access-tchst\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.829736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829677    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f17101f-b5eb-4af2-bc07-a05e07df482e-snapshots\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.829736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829731    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-config\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.829942 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.829789    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-serving-cert\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.830284 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.830263    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-config\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.833431 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.833404    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-serving-cert\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.839341 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.839323    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"]
Sep 09 21:34:05.839408 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.839401    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:05.841818 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.841801    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 21:34:05.841910 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.841791    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f4fvr\""
Sep 09 21:34:05.841910 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.841804    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 21:34:05.841910 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.841900    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 21:34:05.842699 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.842682    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 21:34:05.848489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.848470    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbj8\" (UniqueName: \"kubernetes.io/projected/84dc5d2e-1fbc-4df1-980b-459397d2fbaf-kube-api-access-ksbj8\") pod \"service-ca-operator-859f94c4b-8jtgm\" (UID: \"84dc5d2e-1fbc-4df1-980b-459397d2fbaf\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.930192 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930164    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f17101f-b5eb-4af2-bc07-a05e07df482e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.930343 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930198    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchst\" (UniqueName: \"kubernetes.io/projected/5f17101f-b5eb-4af2-bc07-a05e07df482e-kube-api-access-tchst\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.930343 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930234    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:05.930343 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f17101f-b5eb-4af2-bc07-a05e07df482e-snapshots\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.930500 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930353    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:05.930500 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvh52\" (UniqueName: \"kubernetes.io/projected/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-kube-api-access-hvh52\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:05.930500 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930487    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f17101f-b5eb-4af2-bc07-a05e07df482e-tmp\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.930645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930512    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f17101f-b5eb-4af2-bc07-a05e07df482e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.930645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f17101f-b5eb-4af2-bc07-a05e07df482e-serving-cert\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.930836 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930815    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f17101f-b5eb-4af2-bc07-a05e07df482e-tmp\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.930905 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.930870    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f17101f-b5eb-4af2-bc07-a05e07df482e-snapshots\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.931045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.931025    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f17101f-b5eb-4af2-bc07-a05e07df482e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.931091 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.931044    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f17101f-b5eb-4af2-bc07-a05e07df482e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.932803 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.932784    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f17101f-b5eb-4af2-bc07-a05e07df482e-serving-cert\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.937784 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.937762    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchst\" (UniqueName: \"kubernetes.io/projected/5f17101f-b5eb-4af2-bc07-a05e07df482e-kube-api-access-tchst\") pod \"insights-operator-76bcd6ccf5-ntfn6\" (UID: \"5f17101f-b5eb-4af2-bc07-a05e07df482e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:05.944150 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.944131    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:05.965441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.965415    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"
Sep 09 21:34:05.978569 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:05.978551    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6"
Sep 09 21:34:06.031020 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.030990    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:06.031118 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.031065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:06.031118 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:06.031100    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:06.031200 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.031122    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvh52\" (UniqueName: \"kubernetes.io/projected/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-kube-api-access-hvh52\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:06.031200 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:06.031171    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls podName:5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac nodeName:}" failed. No retries permitted until 2026-09-09 21:34:06.531151214 +0000 UTC m=+48.905475034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ghdpm" (UID: "5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:06.031801 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.031780    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:06.043501 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.043253    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvh52\" (UniqueName: \"kubernetes.io/projected/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-kube-api-access-hvh52\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:06.045510 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.045488    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:06.259197 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.259104    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm"]
Sep 09 21:34:06.262314 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:06.262286    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84dc5d2e_1fbc_4df1_980b_459397d2fbaf.slice/crio-6b34f9506e3f87190e34c54b11680ea428d7ce2ead7b42e288077a477cfd9cc0 WatchSource:0}: Error finding container 6b34f9506e3f87190e34c54b11680ea428d7ce2ead7b42e288077a477cfd9cc0: Status 404 returned error can't find the container with id 6b34f9506e3f87190e34c54b11680ea428d7ce2ead7b42e288077a477cfd9cc0
Sep 09 21:34:06.293862 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.293838    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-ntfn6"]
Sep 09 21:34:06.296855 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:06.296832    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f17101f_b5eb_4af2_bc07_a05e07df482e.slice/crio-2c8ec07ae17a3b8831de5d709fa8a579ddd49f994c377fc5ac5aa3f558244e87 WatchSource:0}: Error finding container 2c8ec07ae17a3b8831de5d709fa8a579ddd49f994c377fc5ac5aa3f558244e87: Status 404 returned error can't find the container with id 2c8ec07ae17a3b8831de5d709fa8a579ddd49f994c377fc5ac5aa3f558244e87
Sep 09 21:34:06.463237 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.463175    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6" event={"ID":"5f17101f-b5eb-4af2-bc07-a05e07df482e","Type":"ContainerStarted","Data":"2c8ec07ae17a3b8831de5d709fa8a579ddd49f994c377fc5ac5aa3f558244e87"}
Sep 09 21:34:06.464299 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.464274    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm" event={"ID":"84dc5d2e-1fbc-4df1-980b-459397d2fbaf","Type":"ContainerStarted","Data":"6b34f9506e3f87190e34c54b11680ea428d7ce2ead7b42e288077a477cfd9cc0"}
Sep 09 21:34:06.534914 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:06.534844    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:06.535049 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:06.535014    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:06.535109 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:06.535094    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls podName:5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac nodeName:}" failed. No retries permitted until 2026-09-09 21:34:07.53507295 +0000 UTC m=+49.909396756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ghdpm" (UID: "5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:07.541673 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:07.541630    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:07.542095 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:07.541800    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:07.542095 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:07.541876    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls podName:5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac nodeName:}" failed. No retries permitted until 2026-09-09 21:34:09.541854714 +0000 UTC m=+51.916178539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ghdpm" (UID: "5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:07.847273 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:07.844926    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:34:07.847273 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:07.845157    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:34:07.847273 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:07.845247    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls podName:b58acc59-65cc-4030-af91-741e1938dc2d nodeName:}" failed. No retries permitted until 2026-09-09 21:34:23.845200291 +0000 UTC m=+66.219524111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls") pod "dns-default-sdxrh" (UID: "b58acc59-65cc-4030-af91-741e1938dc2d") : secret "dns-default-metrics-tls" not found
Sep 09 21:34:08.348753 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:08.348716    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:08.397255 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:08.397222    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:09.558355 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:09.558322    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:09.558681 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:09.558490    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:09.558681 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:09.558547    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls podName:5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac nodeName:}" failed. No retries permitted until 2026-09-09 21:34:13.558526287 +0000 UTC m=+55.932850106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ghdpm" (UID: "5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:10.475370 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.475331    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6" event={"ID":"5f17101f-b5eb-4af2-bc07-a05e07df482e","Type":"ContainerStarted","Data":"520317e94d00a9dd6ecdfea0489b6732741f363bd1736530762abd679ab02cae"}
Sep 09 21:34:10.476910 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.476877    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm" event={"ID":"84dc5d2e-1fbc-4df1-980b-459397d2fbaf","Type":"ContainerStarted","Data":"81a9dea4c77f9518b8b5a19d3078969b99d1e7f24fd4c9d78cb2c16a1cb1257b"}
Sep 09 21:34:10.491003 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.490954    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6" podStartSLOduration=2.334043695 podStartE2EDuration="5.49093743s" totalImagesPullingTime="3.156893735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:10.489991019 +0000 UTC m=+52.864314842" watchObservedRunningTime="2026-09-09 21:34:10.49093743 +0000 UTC m=+52.865261254"
Sep 09 21:34:10.504992 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.504953    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm" podStartSLOduration=2.306695357 podStartE2EDuration="5.504940138s" totalImagesPullingTime="3.198244781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:10.504497031 +0000 UTC m=+52.878820858" watchObservedRunningTime="2026-09-09 21:34:10.504940138 +0000 UTC m=+52.879263962"
Sep 09 21:34:10.823645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.823577    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:10.939399 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.939371    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:10.943944 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.943920    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps"]
Sep 09 21:34:10.967799 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.967773    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps"]
Sep 09 21:34:10.967921 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.967894    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps"
Sep 09 21:34:10.971001 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.970980    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tpslk\""
Sep 09 21:34:10.971614 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.971595    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 21:34:10.971716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:10.971644    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 21:34:11.019188 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.019149    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:11.070770 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.070745    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqglj\" (UniqueName: \"kubernetes.io/projected/905fe570-9725-4238-a36e-9991b7c5ee97-kube-api-access-nqglj\") pod \"migrator-fdbb8fdff-kjxps\" (UID: \"905fe570-9725-4238-a36e-9991b7c5ee97\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps"
Sep 09 21:34:11.081308 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.081249    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:11.172128 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.172087    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqglj\" (UniqueName: \"kubernetes.io/projected/905fe570-9725-4238-a36e-9991b7c5ee97-kube-api-access-nqglj\") pod \"migrator-fdbb8fdff-kjxps\" (UID: \"905fe570-9725-4238-a36e-9991b7c5ee97\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps"
Sep 09 21:34:11.179750 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.179713    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqglj\" (UniqueName: \"kubernetes.io/projected/905fe570-9725-4238-a36e-9991b7c5ee97-kube-api-access-nqglj\") pod \"migrator-fdbb8fdff-kjxps\" (UID: \"905fe570-9725-4238-a36e-9991b7c5ee97\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps"
Sep 09 21:34:11.276895 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.276863    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps"
Sep 09 21:34:11.528819 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.528782    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:11.570155 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.570126    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps"]
Sep 09 21:34:11.574127 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:11.574101    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905fe570_9725_4238_a36e_9991b7c5ee97.slice/crio-8f09180567829cc70c32211773d6c499f7ff105c15ad6ecc1c8d010a372402bd WatchSource:0}: Error finding container 8f09180567829cc70c32211773d6c499f7ff105c15ad6ecc1c8d010a372402bd: Status 404 returned error can't find the container with id 8f09180567829cc70c32211773d6c499f7ff105c15ad6ecc1c8d010a372402bd
Sep 09 21:34:11.605441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.605371    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9f6r8"]
Sep 09 21:34:11.611996 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.611969    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:11.639199 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.639170    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9f6r8"]
Sep 09 21:34:11.639315 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.639288    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.641441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.641411    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 21:34:11.641551 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.641503    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 21:34:11.641551 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.641526    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b8s76\""
Sep 09 21:34:11.671001 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.670979    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:11.776419 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.776391    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.776419 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.776420    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-data-volume\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.776585 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.776441    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.776585 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.776469    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-crio-socket\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.776585 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.776574    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqq28\" (UniqueName: \"kubernetes.io/projected/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-api-access-vqq28\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.877504 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.877432    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-data-volume\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.877504 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.877475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.877504 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.877503    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-crio-socket\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.877927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.877577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqq28\" (UniqueName: \"kubernetes.io/projected/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-api-access-vqq28\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.877927 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:11.877593    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:11.877927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.877656    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.877927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.877673    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-crio-socket\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.877927 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:11.877695    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:12.377673072 +0000 UTC m=+54.751996886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:11.877927 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:11.877744    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:11.877927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.877804    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-data-volume\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:11.877927 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:11.877810    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:12.377794046 +0000 UTC m=+54.752117872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:11.887378 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:11.887358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqq28\" (UniqueName: \"kubernetes.io/projected/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-api-access-vqq28\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:12.381519 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:12.381486    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:12.381519 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:12.381519    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:12.381714 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:12.381615    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:12.381714 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:12.381633    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:12.381714 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:12.381678    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:13.381664543 +0000 UTC m=+55.755988344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:12.381714 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:12.381690    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:13.381684472 +0000 UTC m=+55.756008274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:12.483276 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:12.483241    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps" event={"ID":"905fe570-9725-4238-a36e-9991b7c5ee97","Type":"ContainerStarted","Data":"8f09180567829cc70c32211773d6c499f7ff105c15ad6ecc1c8d010a372402bd"}
Sep 09 21:34:13.123939 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.123905    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-42j84"]
Sep 09 21:34:13.144119 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.144090    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-42j84"]
Sep 09 21:34:13.144264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.144178    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.146268 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.146246    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 21:34:13.146401 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.146315    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 21:34:13.147137 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.147118    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 21:34:13.147271 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.147148    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8j99c\""
Sep 09 21:34:13.147271 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.147157    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 21:34:13.160519 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.160495    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:13.212324 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.212293    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:13.289542 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.289508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rlk\" (UniqueName: \"kubernetes.io/projected/635c8132-be09-4c94-904f-f9181a8de8b2-kube-api-access-b6rlk\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.289685 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.289575    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/635c8132-be09-4c94-904f-f9181a8de8b2-signing-key\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.289685 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.289657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/635c8132-be09-4c94-904f-f9181a8de8b2-signing-cabundle\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.390083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.390003    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:13.390083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.390028    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rlk\" (UniqueName: \"kubernetes.io/projected/635c8132-be09-4c94-904f-f9181a8de8b2-kube-api-access-b6rlk\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.390083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.390049    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:13.390229 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:13.390134    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:13.390229 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:13.390153    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:13.390229 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:13.390200    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:15.39018357 +0000 UTC m=+57.764507404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:13.390369 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:13.390240    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:15.390229888 +0000 UTC m=+57.764553708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:13.390369 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.390287    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/635c8132-be09-4c94-904f-f9181a8de8b2-signing-key\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.390369 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.390336    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/635c8132-be09-4c94-904f-f9181a8de8b2-signing-cabundle\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.390981 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.390962    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/635c8132-be09-4c94-904f-f9181a8de8b2-signing-cabundle\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.392808 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.392787    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/635c8132-be09-4c94-904f-f9181a8de8b2-signing-key\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.398650 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.398629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rlk\" (UniqueName: \"kubernetes.io/projected/635c8132-be09-4c94-904f-f9181a8de8b2-kube-api-access-b6rlk\") pod \"service-ca-75d99c484-42j84\" (UID: \"635c8132-be09-4c94-904f-f9181a8de8b2\") " pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.453099 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.453080    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-42j84"
Sep 09 21:34:13.487911 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.487880    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps" event={"ID":"905fe570-9725-4238-a36e-9991b7c5ee97","Type":"ContainerStarted","Data":"529dd74f9dcfd134503f23d3287181bc601505abfa9579e5f4a60f4218de0cdd"}
Sep 09 21:34:13.592396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.592367    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:13.596025 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:13.592989    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:13.596025 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:13.593068    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls podName:5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac nodeName:}" failed. No retries permitted until 2026-09-09 21:34:21.593049771 +0000 UTC m=+63.967373591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ghdpm" (UID: "5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:13.744310 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:13.744272    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-42j84"]
Sep 09 21:34:13.748391 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:13.748364    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635c8132_be09_4c94_904f_f9181a8de8b2.slice/crio-462f6f71537eae3ce2e3617fae9174b58820ec08a7f8f30cd927454dcdf7399c WatchSource:0}: Error finding container 462f6f71537eae3ce2e3617fae9174b58820ec08a7f8f30cd927454dcdf7399c: Status 404 returned error can't find the container with id 462f6f71537eae3ce2e3617fae9174b58820ec08a7f8f30cd927454dcdf7399c
Sep 09 21:34:14.041318 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:14.041200    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:14.493766 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:14.493721    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps" event={"ID":"905fe570-9725-4238-a36e-9991b7c5ee97","Type":"ContainerStarted","Data":"3a612c52ebc7a098a6a7dcaddb6924cbd4cb98df6d3cdcae88528c3cb535396a"}
Sep 09 21:34:14.495177 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:14.495155    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-42j84" event={"ID":"635c8132-be09-4c94-904f-f9181a8de8b2","Type":"ContainerStarted","Data":"fa19d1d83df44c0d318b8370918f237c2c098cc531a5675995c955de933c6e66"}
Sep 09 21:34:14.495303 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:14.495189    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-42j84" event={"ID":"635c8132-be09-4c94-904f-f9181a8de8b2","Type":"ContainerStarted","Data":"462f6f71537eae3ce2e3617fae9174b58820ec08a7f8f30cd927454dcdf7399c"}
Sep 09 21:34:14.508774 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:14.508731    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kjxps" podStartSLOduration=2.696788016 podStartE2EDuration="4.508716459s" totalImagesPullingTime="1.811928443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:14.507994043 +0000 UTC m=+56.882317867" watchObservedRunningTime="2026-09-09 21:34:14.508716459 +0000 UTC m=+56.883040283"
Sep 09 21:34:14.525487 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:14.525434    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-42j84" podStartSLOduration=1.525413206 podStartE2EDuration="1.525413206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:14.525149001 +0000 UTC m=+56.899472824" watchObservedRunningTime="2026-09-09 21:34:14.525413206 +0000 UTC m=+56.899737029"
Sep 09 21:34:14.569631 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:14.569604    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rnh76_b875a80b-0dd4-4af6-8222-2f86c18b50a8/dns-node-resolver/0.log"
Sep 09 21:34:14.855307 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:14.855224    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:15.030918 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.030883    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:15.133873 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.133757    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:15.170086 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.170058    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k5kh5_8651d356-a086-46cb-9f76-940a0cee764a/node-ca/0.log"
Sep 09 21:34:15.232239 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.232190    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:15.332511 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.332474    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:15.406394 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.406369    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:15.406555 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.406407    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:15.406555 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:15.406490    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:15.406555 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:15.406521    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:15.406682 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:15.406573    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:19.406554326 +0000 UTC m=+61.780878141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:15.406682 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:15.406591    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:19.406583023 +0000 UTC m=+61.780906825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:15.447166 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.447135    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:15.548312 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:15.548278    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:16.133495 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:16.133460    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:16.369913 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:16.369887    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l9tcv_2aee9510-534e-46a2-af83-db4a01b114aa/serve-healthcheck-canary/0.log"
Sep 09 21:34:16.569081 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:16.569058    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kjxps_905fe570-9725-4238-a36e-9991b7c5ee97/migrator/0.log"
Sep 09 21:34:16.770051 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:16.770020    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kjxps_905fe570-9725-4238-a36e-9991b7c5ee97/graceful-termination/0.log"
Sep 09 21:34:16.926001 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:16.925976    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:17.103368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:17.103336    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:17.332635 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:17.332553    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:17.536442 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:17.536413    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:17.742037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:17.742000    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:17.927768 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:17.927728    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:18.415090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:18.415045    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5txw"
Sep 09 21:34:18.425910 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:18.425885    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:18.539137 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:18.539109    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:18.724666 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:18.724587    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:19.130364 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:19.130282    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:19.182304 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:19.182277    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:19.438850 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:19.438824    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:19.438999 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:19.438857    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:19.438999 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:19.438944    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:19.438999 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:19.438953    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:19.439099 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:19.439006    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:27.438991823 +0000 UTC m=+69.813315628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:19.439099 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:19.439028    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm podName:1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:27.439014519 +0000 UTC m=+69.813338325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9f6r8" (UID: "1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:19.530096 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:19.530064    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:19.929405 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:19.929375    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:20.322258 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:20.322161    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:20.423856 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:20.423823    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:20.734970 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:20.734935    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:21.021057 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:21.020978    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:21.131938 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:21.131912    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:21.526772 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:21.526746    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:21.657724 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:21.657695    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:21.657914 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:21.657838    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:21.657977 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:34:21.657922    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls podName:5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac nodeName:}" failed. No retries permitted until 2026-09-09 21:34:37.657905869 +0000 UTC m=+80.032229670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ghdpm" (UID: "5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:34:21.743047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:21.743019    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:22.342701 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:22.342667    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:22.553826 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:22.553791    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:22.937259 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:22.937231    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:23.526967 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.526935    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:23.775057 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.775020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:34:23.775486 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.775165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:34:23.777480 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.777409    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 21:34:23.778105 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.778087    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 21:34:23.788131 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.788114    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2017749-1a67-4077-8ae7-126c4fa84cf2-original-pull-secret\") pod \"global-pull-secret-syncer-mgvxv\" (UID: \"e2017749-1a67-4077-8ae7-126c4fa84cf2\") " pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:34:23.788230 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.788185    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4059f83-eb9e-4c96-b934-0fb98b2716da-metrics-certs\") pod \"network-metrics-daemon-qvhdr\" (UID: \"b4059f83-eb9e-4c96-b934-0fb98b2716da\") " pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:34:23.876365 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.876337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:34:23.878616 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.878590    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b58acc59-65cc-4030-af91-741e1938dc2d-metrics-tls\") pod \"dns-default-sdxrh\" (UID: \"b58acc59-65cc-4030-af91-741e1938dc2d\") " pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:34:23.933264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.933241    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgvxv"
Sep 09 21:34:23.933889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.933867    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:23.948138 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.948122    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5t55x\""
Sep 09 21:34:23.957036 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.957018    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvhdr"
Sep 09 21:34:23.977365 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.977334    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:34:23.980429 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.980410    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 21:34:23.990626 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:23.990343    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 21:34:24.002130 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.002079    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6lp\" (UniqueName: \"kubernetes.io/projected/7a1c2527-af7a-4e75-8e48-530b98632a9e-kube-api-access-hc6lp\") pod \"network-check-target-g6bm5\" (UID: \"7a1c2527-af7a-4e75-8e48-530b98632a9e\") " pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:34:24.035411 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.035232    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:24.091105 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.091076    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4v4f\""
Sep 09 21:34:24.099966 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.099946    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:34:24.212764 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.212737    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mgvxv"]
Sep 09 21:34:24.215734 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:24.215711    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2017749_1a67_4077_8ae7_126c4fa84cf2.slice/crio-0da9f728ffb617907ea06a58329c8b61c5960da81a2d072f0fc4e9062e01b59c WatchSource:0}: Error finding container 0da9f728ffb617907ea06a58329c8b61c5960da81a2d072f0fc4e9062e01b59c: Status 404 returned error can't find the container with id 0da9f728ffb617907ea06a58329c8b61c5960da81a2d072f0fc4e9062e01b59c
Sep 09 21:34:24.239755 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.239730    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvhdr"]
Sep 09 21:34:24.242581 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:24.242552    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4059f83_eb9e_4c96_b934_0fb98b2716da.slice/crio-65a44bd561ee3c841e29d3c59bc7300a481b219662d355ca1a6166653047a33f WatchSource:0}: Error finding container 65a44bd561ee3c841e29d3c59bc7300a481b219662d355ca1a6166653047a33f: Status 404 returned error can't find the container with id 65a44bd561ee3c841e29d3c59bc7300a481b219662d355ca1a6166653047a33f
Sep 09 21:34:24.242716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.242701    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kms6v\""
Sep 09 21:34:24.251330 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.251311    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:34:24.331986 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.331961    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:24.368668 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.368641    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sdxrh"]
Sep 09 21:34:24.371361 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:24.371339    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58acc59_65cc_4030_af91_741e1938dc2d.slice/crio-e485abefa09fbdf5301a26476279fb633e6ad24a40fe5a1adc329a8b335203ba WatchSource:0}: Error finding container e485abefa09fbdf5301a26476279fb633e6ad24a40fe5a1adc329a8b335203ba: Status 404 returned error can't find the container with id e485abefa09fbdf5301a26476279fb633e6ad24a40fe5a1adc329a8b335203ba
Sep 09 21:34:24.530264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.530234    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:24.533040 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.533013    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvhdr" event={"ID":"b4059f83-eb9e-4c96-b934-0fb98b2716da","Type":"ContainerStarted","Data":"65a44bd561ee3c841e29d3c59bc7300a481b219662d355ca1a6166653047a33f"}
Sep 09 21:34:24.534232 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.534194    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mgvxv" event={"ID":"e2017749-1a67-4077-8ae7-126c4fa84cf2","Type":"ContainerStarted","Data":"0da9f728ffb617907ea06a58329c8b61c5960da81a2d072f0fc4e9062e01b59c"}
Sep 09 21:34:24.535448 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.535425    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdxrh" event={"ID":"b58acc59-65cc-4030-af91-741e1938dc2d","Type":"ContainerStarted","Data":"e485abefa09fbdf5301a26476279fb633e6ad24a40fe5a1adc329a8b335203ba"}
Sep 09 21:34:24.542169 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:24.542119    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6bm5"]
Sep 09 21:34:24.545100 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:24.545082    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1c2527_af7a_4e75_8e48_530b98632a9e.slice/crio-a33e92c09ef87c8eea5c054ef2d5d39e4555adb53b1cd5718dba8821dbe70213 WatchSource:0}: Error finding container a33e92c09ef87c8eea5c054ef2d5d39e4555adb53b1cd5718dba8821dbe70213: Status 404 returned error can't find the container with id a33e92c09ef87c8eea5c054ef2d5d39e4555adb53b1cd5718dba8821dbe70213
Sep 09 21:34:25.180071 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:25.180024    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:25.326678 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:25.326599    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:25.533493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:25.533370    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:25.541713 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:25.541679    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6bm5" event={"ID":"7a1c2527-af7a-4e75-8e48-530b98632a9e","Type":"ContainerStarted","Data":"a33e92c09ef87c8eea5c054ef2d5d39e4555adb53b1cd5718dba8821dbe70213"}
Sep 09 21:34:27.506628 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:27.506596    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:27.506628 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:27.506633    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:27.507233 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:27.507193    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:27.509309 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:27.509286    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9f6r8\" (UID: \"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866\") " pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:27.551080 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:27.551052    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b8s76\""
Sep 09 21:34:27.558690 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:27.558671    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9f6r8"
Sep 09 21:34:28.930990 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:28.930955    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:29.297127 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:29.297006    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:29.722173 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:29.721910    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9f6r8"]
Sep 09 21:34:29.726202 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:29.726176    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec642b2_ea6f_44ba_8bf1_0d8c0e9c5866.slice/crio-990c66781f00b9bd7e31bf6368835f4021967fa57a06ea7cf4348c16aa402c7d WatchSource:0}: Error finding container 990c66781f00b9bd7e31bf6368835f4021967fa57a06ea7cf4348c16aa402c7d: Status 404 returned error can't find the container with id 990c66781f00b9bd7e31bf6368835f4021967fa57a06ea7cf4348c16aa402c7d
Sep 09 21:34:30.560757 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.560720    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mgvxv" event={"ID":"e2017749-1a67-4077-8ae7-126c4fa84cf2","Type":"ContainerStarted","Data":"ba5b6aae38a81324616cf0e4921a547727874db96ac53ed66115c69492be099c"}
Sep 09 21:34:30.562672 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.562646    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdxrh" event={"ID":"b58acc59-65cc-4030-af91-741e1938dc2d","Type":"ContainerStarted","Data":"3bc1b3ce8849a2b827aa6707e764bd35159ada29de8afe5e2d04959d67ee209d"}
Sep 09 21:34:30.562797 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.562693    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:34:30.562797 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.562709    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdxrh" event={"ID":"b58acc59-65cc-4030-af91-741e1938dc2d","Type":"ContainerStarted","Data":"450e9d8988efe06834f46e76584a3106ba2d92a7720a45c7d3a308e94572aca9"}
Sep 09 21:34:30.564521 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.564500    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvhdr" event={"ID":"b4059f83-eb9e-4c96-b934-0fb98b2716da","Type":"ContainerStarted","Data":"5ff69f6898dd69d0e43cfb726d67c9541c0c3ee799adffc73f9151375d25c174"}
Sep 09 21:34:30.564626 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.564538    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvhdr" event={"ID":"b4059f83-eb9e-4c96-b934-0fb98b2716da","Type":"ContainerStarted","Data":"b8e87b10230c436c26f093dc85b6ebb280546f90658921a2ed38ae74e34fef50"}
Sep 09 21:34:30.565979 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.565955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6bm5" event={"ID":"7a1c2527-af7a-4e75-8e48-530b98632a9e","Type":"ContainerStarted","Data":"0c634b9075cab9f589e411d72f8a962be76579f57cae5fc502948dfff2799844"}
Sep 09 21:34:30.566083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.566001    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:34:30.567471 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.567446    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9f6r8" event={"ID":"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866","Type":"ContainerStarted","Data":"3f865d5d91dcfb4f3875318e33c4cc9d6b8c95ffbba23659a578c23491b68e0b"}
Sep 09 21:34:30.567560 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.567491    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9f6r8" event={"ID":"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866","Type":"ContainerStarted","Data":"990c66781f00b9bd7e31bf6368835f4021967fa57a06ea7cf4348c16aa402c7d"}
Sep 09 21:34:30.574620 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.574566    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mgvxv" podStartSLOduration=67.332142174 podStartE2EDuration="1m12.574552973s" totalImagesPullingTime="5.242410799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:30.574019251 +0000 UTC m=+72.948343076" watchObservedRunningTime="2026-09-09 21:34:30.574552973 +0000 UTC m=+72.948876796"
Sep 09 21:34:30.588819 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.588783    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sdxrh" podStartSLOduration=34.538029941 podStartE2EDuration="39.588771561s" totalImagesPullingTime="5.05074162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:30.587424421 +0000 UTC m=+72.961748245" watchObservedRunningTime="2026-09-09 21:34:30.588771561 +0000 UTC m=+72.963095385"
Sep 09 21:34:30.600947 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.600913    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qvhdr" podStartSLOduration=67.417033606 podStartE2EDuration="1m12.600902162s" totalImagesPullingTime="5.183868556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:30.600187409 +0000 UTC m=+72.974511234" watchObservedRunningTime="2026-09-09 21:34:30.600902162 +0000 UTC m=+72.975225985"
Sep 09 21:34:30.613848 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:30.613812    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g6bm5" podStartSLOduration=67.699938971 podStartE2EDuration="1m12.613802445s" totalImagesPullingTime="4.913863474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:30.612757565 +0000 UTC m=+72.987081389" watchObservedRunningTime="2026-09-09 21:34:30.613802445 +0000 UTC m=+72.988126269"
Sep 09 21:34:31.572788 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:31.572750    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9f6r8" event={"ID":"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866","Type":"ContainerStarted","Data":"87bd11e798e67792f96d8f89911bbcd99530387298b805bd95c1194446ea4613"}
Sep 09 21:34:32.582479 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:32.582445    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9f6r8" event={"ID":"1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866","Type":"ContainerStarted","Data":"9730a8ca97cebb13ebded71830e6224dbef0c08acb850e981ff6f5e07e1c794f"}
Sep 09 21:34:32.605281 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:32.605234    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9f6r8" podStartSLOduration=19.019068252 podStartE2EDuration="21.605200236s" totalImagesPullingTime="2.586131984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:34:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:32.603756669 +0000 UTC m=+74.978080490" watchObservedRunningTime="2026-09-09 21:34:32.605200236 +0000 UTC m=+74.979524062"
Sep 09 21:34:32.628402 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:32.628372    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:32.729001 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:32.728942    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:32.784064 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:32.784043    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:32.926443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:32.926418    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:33.297147 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:33.297117    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:33.701006 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:33.700980    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:33.802065 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:33.802035    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:34.106726 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.106655    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:34.209117 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.209083    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:34.307723 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.307689    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:34.411465 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.411432    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:34.836029 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.835940    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"]
Sep 09 21:34:34.838285 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.838268    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.841561 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.841529    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 21:34:34.841561 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.841552    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 21:34:34.842376 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.842356    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 21:34:34.842483 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.842422    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 21:34:34.842483 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.842430    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 21:34:34.842604 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.842363    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 21:34:34.842604 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.842428    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 21:34:34.853446 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.853425    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"]
Sep 09 21:34:34.854455 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.854438    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-ca\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.854511 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.854466    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.854511 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.854484    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jxq\" (UniqueName: \"kubernetes.io/projected/65ff859d-9cd9-4e40-affc-a804791f8e56-kube-api-access-97jxq\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.854511 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.854504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-hub\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.854607 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.854587    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.854663 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.854646    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ff859d-9cd9-4e40-affc-a804791f8e56-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.876201 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.876175    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:34.955766 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.955744    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-ca\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.955921 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.955774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.955921 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.955794    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jxq\" (UniqueName: \"kubernetes.io/projected/65ff859d-9cd9-4e40-affc-a804791f8e56-kube-api-access-97jxq\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.955921 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.955815    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-hub\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.955921 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.955867    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.956129 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.955922    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ff859d-9cd9-4e40-affc-a804791f8e56-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.956875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.956847    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ff859d-9cd9-4e40-affc-a804791f8e56-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.958491 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.958466    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-ca\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.958610 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.958520    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-hub\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.958610 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.958548    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.958755 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.958738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65ff859d-9cd9-4e40-affc-a804791f8e56-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.977092 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.977071    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jxq\" (UniqueName: \"kubernetes.io/projected/65ff859d-9cd9-4e40-affc-a804791f8e56-kube-api-access-97jxq\") pod \"cluster-proxy-proxy-agent-c6c6566bb-w8fhx\" (UID: \"65ff859d-9cd9-4e40-affc-a804791f8e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:34.982792 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:34.982769    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:35.088222 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:35.088142    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:35.168760 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:35.168738    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"
Sep 09 21:34:35.188712 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:35.188685    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:35.293503 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:35.293465    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:35.438587 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:35.438555    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx"]
Sep 09 21:34:35.593164 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:35.593129    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx" event={"ID":"65ff859d-9cd9-4e40-affc-a804791f8e56","Type":"ContainerStarted","Data":"aab6c715890de45bcd1d647cee1fbf4aef57466bdd9b1f33422c99fcf5ad3b83"}
Sep 09 21:34:37.677024 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:37.676987    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:37.680112 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:37.680078    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ghdpm\" (UID: \"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:37.950711 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:37.950431    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f4fvr\""
Sep 09 21:34:37.958780 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:37.958753    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"
Sep 09 21:34:39.946349 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:39.946282    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm"]
Sep 09 21:34:39.949408 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:39.949380    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6bbb5c_ca8d_41f8_8d24_944c0848b0ac.slice/crio-3f51cbc81dfc5d9a5aabd87fea12d498316969ddaf92d3c69cbcf753f0da232e WatchSource:0}: Error finding container 3f51cbc81dfc5d9a5aabd87fea12d498316969ddaf92d3c69cbcf753f0da232e: Status 404 returned error can't find the container with id 3f51cbc81dfc5d9a5aabd87fea12d498316969ddaf92d3c69cbcf753f0da232e
Sep 09 21:34:40.575623 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:40.575591    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sdxrh"
Sep 09 21:34:40.613546 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:40.613509    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx" event={"ID":"65ff859d-9cd9-4e40-affc-a804791f8e56","Type":"ContainerStarted","Data":"c0265804e874c3358e6488d02ee13b041b8887c622ad50497e0faf7643373207"}
Sep 09 21:34:40.613708 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:40.613561    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx" event={"ID":"65ff859d-9cd9-4e40-affc-a804791f8e56","Type":"ContainerStarted","Data":"18cfab845a5fa9888e0b2212cdecefb3edb627f7acdd0bf67ac8f0182279e0bc"}
Sep 09 21:34:40.613708 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:40.613579    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx" event={"ID":"65ff859d-9cd9-4e40-affc-a804791f8e56","Type":"ContainerStarted","Data":"8925b1e72d943b2ffee2b448d6589f47f7f653f78cf0bdfa64346ce86dceacb0"}
Sep 09 21:34:40.614797 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:40.614757    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm" event={"ID":"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac","Type":"ContainerStarted","Data":"3f51cbc81dfc5d9a5aabd87fea12d498316969ddaf92d3c69cbcf753f0da232e"}
Sep 09 21:34:40.630539 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:40.630495    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c6c6566bb-w8fhx" podStartSLOduration=2.341317944 podStartE2EDuration="6.630480118s" totalImagesPullingTime="4.289162174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:40.630042012 +0000 UTC m=+83.004365836" watchObservedRunningTime="2026-09-09 21:34:40.630480118 +0000 UTC m=+83.004803941"
Sep 09 21:34:42.199021 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:42.198979    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:42.248570 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:42.248500    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:42.625681 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:42.625599    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm" event={"ID":"5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac","Type":"ContainerStarted","Data":"021093865cf289f5b5a6a32e9bce70d99213622cf8c8e3b1cc81f71871c51317"}
Sep 09 21:34:42.628482 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:42.628454    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:42.642255 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:42.642194    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ghdpm" podStartSLOduration=35.273893726 podStartE2EDuration="37.642180695s" totalImagesPullingTime="2.368286969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:42.641794905 +0000 UTC m=+85.016118730" watchObservedRunningTime="2026-09-09 21:34:42.642180695 +0000 UTC m=+85.016504519"
Sep 09 21:34:42.725883 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:42.725859    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:42.800629 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:42.800598    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:43.114812 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:43.114780    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:45.530833 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:45.530804    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:45.585661 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:45.585619    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:46.125534 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:46.125488    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:46.173859 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:46.173831    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:47.807334 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:47.807299    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:47.867464 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:47.867435    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:48.190402 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:48.190367    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:49.153472 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:49.153410    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:49.205253 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:49.205222    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:49.325826 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:49.325790    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:49.426768 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:49.426682    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:50.694063 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:50.694032    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:52.331183 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:52.331139    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:52.796482 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:52.796451    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:52.970570 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:52.970539    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:53.139582 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.139506    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:53.218417 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.218383    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nxs2p"]
Sep 09 21:34:53.221587 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.221566    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.223966 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.223937    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9tb2f\""
Sep 09 21:34:53.224091 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.224020    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 21:34:53.224150 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.223941    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 21:34:53.224754 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.224737    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 21:34:53.224823 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.224766    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 21:34:53.261864 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.261836    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:53.283880 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.283850    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6wk\" (UniqueName: \"kubernetes.io/projected/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-kube-api-access-qt6wk\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.283990 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.283911    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.283990 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.283942    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-textfile\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.283990 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.283970    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-sys\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.284108 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.283995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-tls\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.284108 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.284019    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-root\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.284108 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.284041    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-metrics-client-ca\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.284108 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.284069    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-wtmp\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.284108 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.284091    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.348423 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.348391    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:53.385255 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385227    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-wtmp\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385268    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385312    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6wk\" (UniqueName: \"kubernetes.io/projected/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-kube-api-access-qt6wk\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385365    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385379    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-wtmp\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385392    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-textfile\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385641 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385431    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-sys\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385641 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385460    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-tls\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385641 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385486    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-root\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385641 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385507    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-metrics-client-ca\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385641 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385526    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-sys\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385845 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385711    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-root\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385845 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385725    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-textfile\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.385985 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.385967    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.386041 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.386031    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-metrics-client-ca\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.387798 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.387779    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.387977 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.387956    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-node-exporter-tls\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.394470 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.394418    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6wk\" (UniqueName: \"kubernetes.io/projected/eacb2e6e-8c75-4f97-a7eb-297d04c695ac-kube-api-access-qt6wk\") pod \"node-exporter-nxs2p\" (UID: \"eacb2e6e-8c75-4f97-a7eb-297d04c695ac\") " pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.442045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.442026    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:53.532989 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.532963    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nxs2p"
Sep 09 21:34:53.543670 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:34:53.543642    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacb2e6e_8c75_4f97_a7eb_297d04c695ac.slice/crio-725a43da58c8b329d21716eb1f99b6f39604fc9fb98d3935566091355ea0be4f WatchSource:0}: Error finding container 725a43da58c8b329d21716eb1f99b6f39604fc9fb98d3935566091355ea0be4f: Status 404 returned error can't find the container with id 725a43da58c8b329d21716eb1f99b6f39604fc9fb98d3935566091355ea0be4f
Sep 09 21:34:53.664442 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:53.664412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxs2p" event={"ID":"eacb2e6e-8c75-4f97-a7eb-297d04c695ac","Type":"ContainerStarted","Data":"725a43da58c8b329d21716eb1f99b6f39604fc9fb98d3935566091355ea0be4f"}
Sep 09 21:34:54.244789 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:54.244752    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:54.320268 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:54.320231    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:54.450273 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:54.450234    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:54.545998 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:54.545936    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:55.673981 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:55.673945    2585 generic.go:356] "Generic (PLEG): container finished" podID="eacb2e6e-8c75-4f97-a7eb-297d04c695ac" containerID="879aa30e2186bc77e3fc88c5be551be654ff08a8b637e4e0adcc6b9f9ee004c1" exitCode=0
Sep 09 21:34:55.674424 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:55.673955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxs2p" event={"ID":"eacb2e6e-8c75-4f97-a7eb-297d04c695ac","Type":"ContainerDied","Data":"879aa30e2186bc77e3fc88c5be551be654ff08a8b637e4e0adcc6b9f9ee004c1"}
Sep 09 21:34:56.158513 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:56.158486    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:56.258853 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:56.258816    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:56.568141 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:56.568102    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:56.679956 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:56.679925    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxs2p" event={"ID":"eacb2e6e-8c75-4f97-a7eb-297d04c695ac","Type":"ContainerStarted","Data":"0a57ad842286bf65485a6140d9fafac220e5fd6c41c4cdd49e7e1519970baba3"}
Sep 09 21:34:56.680326 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:56.679974    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxs2p" event={"ID":"eacb2e6e-8c75-4f97-a7eb-297d04c695ac","Type":"ContainerStarted","Data":"9ae68b145f5661239bfe77378e5230df65a4217e71b713b9348930ad92368153"}
Sep 09 21:34:56.699046 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:56.699003    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nxs2p" podStartSLOduration=2.397969367 podStartE2EDuration="3.698987812s" totalImagesPullingTime="1.229490546s" totalInitContainerRuntime="71.527899ms" isStatefulPod=true podCreationTimestamp="2026-09-09 21:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:56.697263078 +0000 UTC m=+99.071586902" watchObservedRunningTime="2026-09-09 21:34:56.698987812 +0000 UTC m=+99.073311652"
Sep 09 21:34:56.767079 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:56.767040    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:57.778185 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:57.778143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:57.878615 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:57.878585    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:57.987345 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:57.987317    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.045860 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.045795    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.158086 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.158062    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.219466 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.219436    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.274881 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.274855    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.608849 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.608823    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.708938 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.708907    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.809404 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.809373    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.937973 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.937938    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:58.993844 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:58.993811    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:59.414470 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.414441    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 21:34:59.418258 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.418240    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.420672 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.420644    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 21:34:59.420833 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.420663    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 21:34:59.420833 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.420671    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 21:34:59.420833 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.420817    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 21:34:59.420833 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.420701    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 21:34:59.421080 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.420700    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 21:34:59.421181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.421136    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 21:34:59.421271 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.421262    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9tpa4cpmtphe3\""
Sep 09 21:34:59.421392 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.421369    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zlvp8\""
Sep 09 21:34:59.421392 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.421387    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 21:34:59.421508 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.421430    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 21:34:59.421508 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.421473    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 21:34:59.422016 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.421998    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 21:34:59.422137 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.422096    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 21:34:59.422480 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.422458    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 21:34:59.424257 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.424240    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 21:34:59.429765 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.429743    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 21:34:59.436605 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.436585    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 21:34:59.439425 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.439400    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:59.533504 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533472    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.533504 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533507    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.533716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533529    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.533716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533551    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.533716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533628    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xcb\" (UniqueName: \"kubernetes.io/projected/c0f5225f-c0ed-444c-8c09-007a3c47448a-kube-api-access-29xcb\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.533716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533653    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.533716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.533716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533689    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.533716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533710    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533775    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0f5225f-c0ed-444c-8c09-007a3c47448a-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533828    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-config\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533854    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0f5225f-c0ed-444c-8c09-007a3c47448a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533877    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533912    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533949    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533974    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.533997    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534047 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.534040    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534351 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.534064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.534351 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.534089    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.634771 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.634741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0f5225f-c0ed-444c-8c09-007a3c47448a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.634771 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.634774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.634951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.634802    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.634951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.634835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.634951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.634858    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.634951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.634880    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.634951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.634920    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635224 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635182    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635290 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635347 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635285    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635347 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635312    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635451 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635348    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635451 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635372    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635451 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635407    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xcb\" (UniqueName: \"kubernetes.io/projected/c0f5225f-c0ed-444c-8c09-007a3c47448a-kube-api-access-29xcb\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635451 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635434    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635461    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635489    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635517    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635547    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0f5225f-c0ed-444c-8c09-007a3c47448a-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-config\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.635640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.635600    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.636343 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.636315    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.636660 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.636617    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.637820 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.637794    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.638143 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.638115    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0f5225f-c0ed-444c-8c09-007a3c47448a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.638888 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.638540    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.639720 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.639693    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.639829 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.639744    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.639985 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.639948    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.640226 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.640160    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.640226 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.640177    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.640372 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.640256    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.640800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.640781    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.640973 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.640948    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0f5225f-c0ed-444c-8c09-007a3c47448a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.641057 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.641009    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-config\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.641243 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.641180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0f5225f-c0ed-444c-8c09-007a3c47448a-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.642250 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.642232    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.642417 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.642394    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.642497 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.642473    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0f5225f-c0ed-444c-8c09-007a3c47448a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.648525 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.648504    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xcb\" (UniqueName: \"kubernetes.io/projected/c0f5225f-c0ed-444c-8c09-007a3c47448a-kube-api-access-29xcb\") pod \"prometheus-k8s-0\" (UID: \"c0f5225f-c0ed-444c-8c09-007a3c47448a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:34:59.728392 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:34:59.728339    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:35:00.011567 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:00.011481    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 21:35:00.015680 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:35:00.015649    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f5225f_c0ed_444c_8c09_007a3c47448a.slice/crio-36bf41b4bd2ba12e3e2e3b746bcb939d225dfc8f35ea86ed1b5aeaf8507edce3 WatchSource:0}: Error finding container 36bf41b4bd2ba12e3e2e3b746bcb939d225dfc8f35ea86ed1b5aeaf8507edce3: Status 404 returned error can't find the container with id 36bf41b4bd2ba12e3e2e3b746bcb939d225dfc8f35ea86ed1b5aeaf8507edce3
Sep 09 21:35:00.171412 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:00.171378    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:00.222405 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:00.222375    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:00.694187 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:00.694149    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0f5225f-c0ed-444c-8c09-007a3c47448a","Type":"ContainerStarted","Data":"36bf41b4bd2ba12e3e2e3b746bcb939d225dfc8f35ea86ed1b5aeaf8507edce3"}
Sep 09 21:35:01.575651 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:01.575621    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g6bm5"
Sep 09 21:35:02.707372 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:02.707341    2585 generic.go:356] "Generic (PLEG): container finished" podID="c0f5225f-c0ed-444c-8c09-007a3c47448a" containerID="e4dd6c80fe35cbccf3719c17818134c743f7be333022664fdeb9e368af3c1e3d" exitCode=0
Sep 09 21:35:02.707831 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:02.707426    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0f5225f-c0ed-444c-8c09-007a3c47448a","Type":"ContainerDied","Data":"e4dd6c80fe35cbccf3719c17818134c743f7be333022664fdeb9e368af3c1e3d"}
Sep 09 21:35:03.536565 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:03.536508    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:03.586718 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:03.586693    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:08.730912 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:08.730877    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0f5225f-c0ed-444c-8c09-007a3c47448a","Type":"ContainerStarted","Data":"74cabf80f5db74552a60b0738277b39586c2adc4622dd60bbbf638b09c67692a"}
Sep 09 21:35:08.731297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:08.730940    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0f5225f-c0ed-444c-8c09-007a3c47448a","Type":"ContainerStarted","Data":"c90f33620c292f6034f166267554b12c584fb993af4eb98e50653210a95b6eef"}
Sep 09 21:35:09.563917 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:09.563881    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:09.664153 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:09.664091    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:09.718130 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:09.718094    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:10.742040 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:10.742008    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0f5225f-c0ed-444c-8c09-007a3c47448a","Type":"ContainerStarted","Data":"7482811d85c6781289b8e802a0a053bd703bec494705d5c11c3728f2497f7c6b"}
Sep 09 21:35:10.742484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:10.742070    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0f5225f-c0ed-444c-8c09-007a3c47448a","Type":"ContainerStarted","Data":"82c0b29bb14ade30902a44e6875e2e5e1d4a2721afecd568debd9bc981d98664"}
Sep 09 21:35:10.742484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:10.742096    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0f5225f-c0ed-444c-8c09-007a3c47448a","Type":"ContainerStarted","Data":"427608b81ee071a9a90edd85b61f175c8ccec1611c5c9037917da8dcd122d933"}
Sep 09 21:35:10.742484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:10.742121    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0f5225f-c0ed-444c-8c09-007a3c47448a","Type":"ContainerStarted","Data":"c05044b74ac7cbd5c53663094a7110a0ff90a9bee778586e14abe1533f3c4583"}
Sep 09 21:35:10.775032 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:10.774978    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.779749892 podStartE2EDuration="11.774955955s" totalImagesPullingTime="8.980758065s" totalInitContainerRuntime="14.447998ms" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:10.773009694 +0000 UTC m=+113.147333531" watchObservedRunningTime="2026-09-09 21:35:10.774955955 +0000 UTC m=+113.149279781"
Sep 09 21:35:10.864809 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:10.864781    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:10.968123 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:10.968091    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:11.065504 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:11.065443    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:11.116901 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:11.116877    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:13.666878 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:13.666849    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:13.767318 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:13.767282    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:13.933104 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:13.933026    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:14.087459 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:14.087428    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:14.729316 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:14.729267    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:35:26.212997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:26.212945    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:27.850604 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:27.850569    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:27.951288 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:27.951250    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:30.818196 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:30.818162    2585 generic.go:356] "Generic (PLEG): container finished" podID="84dc5d2e-1fbc-4df1-980b-459397d2fbaf" containerID="81a9dea4c77f9518b8b5a19d3078969b99d1e7f24fd4c9d78cb2c16a1cb1257b" exitCode=0
Sep 09 21:35:30.818574 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:30.818229    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm" event={"ID":"84dc5d2e-1fbc-4df1-980b-459397d2fbaf","Type":"ContainerDied","Data":"81a9dea4c77f9518b8b5a19d3078969b99d1e7f24fd4c9d78cb2c16a1cb1257b"}
Sep 09 21:35:30.818574 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:30.818522    2585 scope.go:117] "RemoveContainer" containerID="81a9dea4c77f9518b8b5a19d3078969b99d1e7f24fd4c9d78cb2c16a1cb1257b"
Sep 09 21:35:31.823727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:31.823688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jtgm" event={"ID":"84dc5d2e-1fbc-4df1-980b-459397d2fbaf","Type":"ContainerStarted","Data":"5ed6a35f29b844fa92fd295486d0764de91029880e97280d9b318cac734df0cf"}
Sep 09 21:35:41.819847 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:41.819812    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:41.870606 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:41.870581    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:45.873714 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:45.873678    2585 generic.go:356] "Generic (PLEG): container finished" podID="5f17101f-b5eb-4af2-bc07-a05e07df482e" containerID="520317e94d00a9dd6ecdfea0489b6732741f363bd1736530762abd679ab02cae" exitCode=0
Sep 09 21:35:45.874129 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:45.873754    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6" event={"ID":"5f17101f-b5eb-4af2-bc07-a05e07df482e","Type":"ContainerDied","Data":"520317e94d00a9dd6ecdfea0489b6732741f363bd1736530762abd679ab02cae"}
Sep 09 21:35:45.874129 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:45.874088    2585 scope.go:117] "RemoveContainer" containerID="520317e94d00a9dd6ecdfea0489b6732741f363bd1736530762abd679ab02cae"
Sep 09 21:35:46.880928 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:46.880896    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-ntfn6" event={"ID":"5f17101f-b5eb-4af2-bc07-a05e07df482e","Type":"ContainerStarted","Data":"9a59947d934ddc373fa42c1b49ba3dfcf5ffd0084af13f7a13d4feacaec190ad"}
Sep 09 21:35:46.970297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:46.970268    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:47.064800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:47.064767    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:59.729172 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:59.729138    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:35:59.752166 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:59.752140    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:35:59.948691 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:35:59.948660    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:36:12.428063 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:12.427972    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:12.629520 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:12.629491    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:21.126940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:21.126881    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:21.184324 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:21.184289    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:21.267717 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:21.267667    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:21.322547 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:21.322519    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:23.002026 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:23.001995    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:23.053142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:23.053109    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:27.627964 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:27.627929    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:27.676443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:27.676411    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:32.504268 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:32.504231    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:32.604641 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:32.604609    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:33.114803 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:33.114768    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:33.179361 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:33.179331    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:39.552662 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:39.552623    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:39.604360 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:39.604330    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:44.549971 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:44.549934    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:44.603702 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:44.603666    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:51.030073 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:36:51.030021    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:15.498061 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:15.498025    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:15.590851 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:15.589441    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:17.021499 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:17.021464    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:42.020541 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:42.020458    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:43.405268 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:43.405201    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:45.630201 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:45.630165    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:45.730117 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:45.730065    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:47.575461 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:47.575429    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:47.687862 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:47.687836    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:47.897936 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:47.897859    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:47.995571 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:47.995540    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:50.265353 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:50.265320    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:50.473441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:50.473385    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:52.812729 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:52.812672    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:52.911459 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:52.911425    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:56.625543 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:56.625500    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:56.724735 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:56.724700    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:57.861683 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:57.861654    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:59.207455 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:59.207426    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:59.307807 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:37:59.307774    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:03.096575 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:03.096532    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:03.197059 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:03.197015    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:04.972819 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:04.972774    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:17.983152 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:17.983118    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:38:17.989026 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:17.989003    2585 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 21:38:17.993069 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:17.993047    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:38:19.250642 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:19.250609    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:19.326038 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:19.326009    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:20.620060 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:20.620024    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:20.713969 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:20.713943    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:21.234291 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.234260    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:21.445300 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.445271    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:21.542181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.542096    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:21.684956 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.684922    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"]
Sep 09 21:38:21.688074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.688053    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.690373 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.690347    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 21:38:21.690737 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.690719    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 21:38:21.690824 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.690764    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 21:38:21.690883 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.690821    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 21:38:21.690883 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.690820    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9n6ch\""
Sep 09 21:38:21.708798 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.708779    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"]
Sep 09 21:38:21.788913 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.788883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/844f6c61-76af-4b00-a030-49d052aa4cd8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.789048 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.788918    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/844f6c61-76af-4b00-a030-49d052aa4cd8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.789048 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.788992    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bfrn\" (UniqueName: \"kubernetes.io/projected/844f6c61-76af-4b00-a030-49d052aa4cd8-kube-api-access-5bfrn\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.789048 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.789025    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/844f6c61-76af-4b00-a030-49d052aa4cd8-manifests\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.789172 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.789064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/844f6c61-76af-4b00-a030-49d052aa4cd8-tmp\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.889900 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.889827    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bfrn\" (UniqueName: \"kubernetes.io/projected/844f6c61-76af-4b00-a030-49d052aa4cd8-kube-api-access-5bfrn\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.889900 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.889862    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/844f6c61-76af-4b00-a030-49d052aa4cd8-manifests\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.889900 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.889894    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/844f6c61-76af-4b00-a030-49d052aa4cd8-tmp\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.890145 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.889938    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/844f6c61-76af-4b00-a030-49d052aa4cd8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.890145 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.889959    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/844f6c61-76af-4b00-a030-49d052aa4cd8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.890267 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.890247    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/844f6c61-76af-4b00-a030-49d052aa4cd8-manifests\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.892438 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.892412    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/844f6c61-76af-4b00-a030-49d052aa4cd8-tmp\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.892551 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.892533    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/844f6c61-76af-4b00-a030-49d052aa4cd8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.892732 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.892713    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/844f6c61-76af-4b00-a030-49d052aa4cd8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.898173 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.898144    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bfrn\" (UniqueName: \"kubernetes.io/projected/844f6c61-76af-4b00-a030-49d052aa4cd8-kube-api-access-5bfrn\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-xwsjp\" (UID: \"844f6c61-76af-4b00-a030-49d052aa4cd8\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:21.998201 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:21.998155    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:22.285965 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:22.285932    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"]
Sep 09 21:38:22.289189 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:38:22.289159    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844f6c61_76af_4b00_a030_49d052aa4cd8.slice/crio-f66f239f0e8329c801be5009740188f3afa0347d6d5f22569111f4de08e1054b WatchSource:0}: Error finding container f66f239f0e8329c801be5009740188f3afa0347d6d5f22569111f4de08e1054b: Status 404 returned error can't find the container with id f66f239f0e8329c801be5009740188f3afa0347d6d5f22569111f4de08e1054b
Sep 09 21:38:22.291028 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:22.291010    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:38:22.429073 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:22.429042    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp" event={"ID":"844f6c61-76af-4b00-a030-49d052aa4cd8","Type":"ContainerStarted","Data":"f66f239f0e8329c801be5009740188f3afa0347d6d5f22569111f4de08e1054b"}
Sep 09 21:38:22.563907 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:22.563811    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:22.668413 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:22.668380    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:26.447742 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:26.447706    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp" event={"ID":"844f6c61-76af-4b00-a030-49d052aa4cd8","Type":"ContainerStarted","Data":"7f605dd673217b9dc54b79ebb2afa8bd1bf4ee24b51537ecad8ed45a143b314a"}
Sep 09 21:38:27.453282 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:27.453248    2585 generic.go:356] "Generic (PLEG): container finished" podID="844f6c61-76af-4b00-a030-49d052aa4cd8" containerID="7f605dd673217b9dc54b79ebb2afa8bd1bf4ee24b51537ecad8ed45a143b314a" exitCode=0
Sep 09 21:38:27.453282 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:27.453286    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp" event={"ID":"844f6c61-76af-4b00-a030-49d052aa4cd8","Type":"ContainerDied","Data":"7f605dd673217b9dc54b79ebb2afa8bd1bf4ee24b51537ecad8ed45a143b314a"}
Sep 09 21:38:28.459068 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:28.459034    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp" event={"ID":"844f6c61-76af-4b00-a030-49d052aa4cd8","Type":"ContainerStarted","Data":"e0ec12c0645de21fe0fcf38a7122e123299888b72eb9b3f214f2732f3f4522ff"}
Sep 09 21:38:28.459459 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:28.459172    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:28.487433 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:28.487379    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp" podStartSLOduration=2.979361083 podStartE2EDuration="7.487325206s" totalImagesPullingTime="3.505902975s" totalInitContainerRuntime="1.002061148s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:28.486308562 +0000 UTC m=+310.860632385" watchObservedRunningTime="2026-09-09 21:38:28.487325206 +0000 UTC m=+310.861649030"
Sep 09 21:38:33.641114 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:33.641076    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:33.741717 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:33.741684    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:39.464899 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:39.464872    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-xwsjp"
Sep 09 21:38:40.970698 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:40.970667    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:41.356799 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.356724    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5769888876-m84dl"]
Sep 09 21:38:41.359632 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.359617    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.365888 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.365852    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 21:38:41.365979 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.365863    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 21:38:41.366037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.366017    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 21:38:41.366269 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.366251    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 21:38:41.382941 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.382920    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5769888876-m84dl"]
Sep 09 21:38:41.452530 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.452505    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxr67\" (UniqueName: \"kubernetes.io/projected/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-kube-api-access-nxr67\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.452658 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.452556    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-tmp\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.452658 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.452590    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-tls-certs\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.553249 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.553201    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-tmp\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.553405 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.553256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-tls-certs\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.553405 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.553296    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxr67\" (UniqueName: \"kubernetes.io/projected/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-kube-api-access-nxr67\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.555681 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.555659    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-tmp\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.555872 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.555851    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-tls-certs\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.564609 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.564583    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxr67\" (UniqueName: \"kubernetes.io/projected/50d49099-98f8-4bfe-bdce-4a90f8abfcb7-kube-api-access-nxr67\") pod \"kube-auth-proxy-5769888876-m84dl\" (UID: \"50d49099-98f8-4bfe-bdce-4a90f8abfcb7\") " pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.668321 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.668300    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl"
Sep 09 21:38:41.938749 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:41.938673    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5769888876-m84dl"]
Sep 09 21:38:41.941505 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:38:41.941478    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d49099_98f8_4bfe_bdce_4a90f8abfcb7.slice/crio-33970220dbf6da458f4147ce15b3d6d15e04ddc5b5a4a1b46499719ffe04a2d4 WatchSource:0}: Error finding container 33970220dbf6da458f4147ce15b3d6d15e04ddc5b5a4a1b46499719ffe04a2d4: Status 404 returned error can't find the container with id 33970220dbf6da458f4147ce15b3d6d15e04ddc5b5a4a1b46499719ffe04a2d4
Sep 09 21:38:42.515276 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.515242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl" event={"ID":"50d49099-98f8-4bfe-bdce-4a90f8abfcb7","Type":"ContainerStarted","Data":"33970220dbf6da458f4147ce15b3d6d15e04ddc5b5a4a1b46499719ffe04a2d4"}
Sep 09 21:38:42.711461 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.711427    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"]
Sep 09 21:38:42.715472 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.715456    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.717598 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.717576    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 21:38:42.717749 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.717727    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:38:42.717831 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.717770    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 21:38:42.717831 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.717803    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6r8r6\""
Sep 09 21:38:42.718031 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.718017    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 21:38:42.727838 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.727820    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"]
Sep 09 21:38:42.864680 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.864596    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbkj\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-kube-api-access-rpbkj\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.864680 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.864631    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.864680 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.864659    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.864894 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.864713    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f5a126-f044-449f-b0e4-1c89b58d8c8b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.864894 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.864728    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.864894 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.864778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.864894 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.864806    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.965997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.965960    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.966181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.966020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbkj\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-kube-api-access-rpbkj\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.966181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.966049    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.966181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.966088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.966181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.966121    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f5a126-f044-449f-b0e4-1c89b58d8c8b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.966181 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.966144    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.966469 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.966187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.966827 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.966758    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.968582 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.968563    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.968687 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.968667    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.968962 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.968942    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f5a126-f044-449f-b0e4-1c89b58d8c8b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.969080 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.969063    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.973709 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.973687    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:42.973955 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:42.973939    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbkj\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-kube-api-access-rpbkj\") pod \"istiod-openshift-gateway-7b7995d5cd-xm4k6\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:43.024990 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:43.024944    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:43.188582 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:43.188547    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:43.300444 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:43.300382    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"]
Sep 09 21:38:43.303753 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:38:43.303708    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f5a126_f044_449f_b0e4_1c89b58d8c8b.slice/crio-d3e5f61d5c6955f27b5c9a71be858db2bcd4941752f2f87ef57cff8df6cb397a WatchSource:0}: Error finding container d3e5f61d5c6955f27b5c9a71be858db2bcd4941752f2f87ef57cff8df6cb397a: Status 404 returned error can't find the container with id d3e5f61d5c6955f27b5c9a71be858db2bcd4941752f2f87ef57cff8df6cb397a
Sep 09 21:38:43.523536 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:43.523447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6" event={"ID":"79f5a126-f044-449f-b0e4-1c89b58d8c8b","Type":"ContainerStarted","Data":"d3e5f61d5c6955f27b5c9a71be858db2bcd4941752f2f87ef57cff8df6cb397a"}
Sep 09 21:38:43.727375 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:43.727318    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:45.534835 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:45.534798    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl" event={"ID":"50d49099-98f8-4bfe-bdce-4a90f8abfcb7","Type":"ContainerStarted","Data":"4f3faeb28e7245b66939b5c51c30c1cf24a0672a779cbe5254bd6e9c5eac9e3f"}
Sep 09 21:38:45.550536 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:45.550445    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5769888876-m84dl" podStartSLOduration=1.089672693 podStartE2EDuration="4.550424784s" totalImagesPullingTime="3.460752091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:45.548551801 +0000 UTC m=+327.922875626" watchObservedRunningTime="2026-09-09 21:38:45.550424784 +0000 UTC m=+327.924748609"
Sep 09 21:38:47.273607 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:47.273556    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 21:38:47.273882 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:47.273636    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 21:38:47.545362 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:47.545274    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6" event={"ID":"79f5a126-f044-449f-b0e4-1c89b58d8c8b","Type":"ContainerStarted","Data":"044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6"}
Sep 09 21:38:47.545518 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:47.545359    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:47.563588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:47.563532    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6" podStartSLOduration=1.59647243 podStartE2EDuration="5.563517282s" totalImagesPullingTime="3.967044852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:47.562244338 +0000 UTC m=+329.936568162" watchObservedRunningTime="2026-09-09 21:38:47.563517282 +0000 UTC m=+329.937841105"
Sep 09 21:38:47.942649 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:47.942616    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:48.042642 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:48.042609    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:48.411378 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:48.411331    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:48.551160 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:48.551135    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:38:48.585814 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:48.585786    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:48.653677 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:48.653642    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:52.036664 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:52.036629    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:52.095028 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:52.094990    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:52.195723 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:52.195697    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:52.295927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:52.295856    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:52.576517 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:52.576446    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:53.081274 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:53.081238    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:59.595185 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.595154    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"]
Sep 09 21:38:59.598490 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.598473    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.606626 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.606599    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.606748 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.606658    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34e697-7c3d-4685-9636-b6134b5ba4f5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.606814 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.606766    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.606870 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.606830    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.606870 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.606862    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbz8\" (UniqueName: \"kubernetes.io/projected/dc34e697-7c3d-4685-9636-b6134b5ba4f5-kube-api-access-cgbz8\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.606972 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.606894    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.606972 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.606932    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.608797 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.608773    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"]
Sep 09 21:38:59.707765 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.707726    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.707765 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.707763    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbz8\" (UniqueName: \"kubernetes.io/projected/dc34e697-7c3d-4685-9636-b6134b5ba4f5-kube-api-access-cgbz8\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.707997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.707791    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.707997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.707828    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.707997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.707875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.707997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.707917    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34e697-7c3d-4685-9636-b6134b5ba4f5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.707997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.707968    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.708524 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.708501    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.710429 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.710400    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34e697-7c3d-4685-9636-b6134b5ba4f5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.710524 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.710496    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.710656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.710638    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.710805 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.710786    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.714888 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.714869    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34e697-7c3d-4685-9636-b6134b5ba4f5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.715368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.715346    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbz8\" (UniqueName: \"kubernetes.io/projected/dc34e697-7c3d-4685-9636-b6134b5ba4f5-kube-api-access-cgbz8\") pod \"istiod-openshift-gateway-5677ccb49b-xzdss\" (UID: \"dc34e697-7c3d-4685-9636-b6134b5ba4f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:38:59.908201 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:38:59.908172    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:39:00.213546 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:00.213474    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"]
Sep 09 21:39:00.219001 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:39:00.218978    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc34e697_7c3d_4685_9636_b6134b5ba4f5.slice/crio-766b6b3317f925b0bc1538b8e7d9c9a84c8a56c03a3291fc2412899d74995003 WatchSource:0}: Error finding container 766b6b3317f925b0bc1538b8e7d9c9a84c8a56c03a3291fc2412899d74995003: Status 404 returned error can't find the container with id 766b6b3317f925b0bc1538b8e7d9c9a84c8a56c03a3291fc2412899d74995003
Sep 09 21:39:00.221301 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:00.221266    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 21:39:00.221401 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:00.221325    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 21:39:00.604291 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:00.604169    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss" event={"ID":"dc34e697-7c3d-4685-9636-b6134b5ba4f5","Type":"ContainerStarted","Data":"be605827f0b9974fdeb12c4b97a70b7c67b4f179699e265054f2cc914268a359"}
Sep 09 21:39:00.604291 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:00.604258    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss" event={"ID":"dc34e697-7c3d-4685-9636-b6134b5ba4f5","Type":"ContainerStarted","Data":"766b6b3317f925b0bc1538b8e7d9c9a84c8a56c03a3291fc2412899d74995003"}
Sep 09 21:39:00.604291 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:00.604286    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:39:00.655074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:00.655020    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss" podStartSLOduration=1.655006612 podStartE2EDuration="1.655006612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:00.653800615 +0000 UTC m=+343.028124634" watchObservedRunningTime="2026-09-09 21:39:00.655006612 +0000 UTC m=+343.029330429"
Sep 09 21:39:01.610609 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:01.610583    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzdss"
Sep 09 21:39:01.724942 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:01.724911    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"]
Sep 09 21:39:01.725187 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:01.725161    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6" podUID="79f5a126-f044-449f-b0e4-1c89b58d8c8b" containerName="discovery" containerID="cri-o://044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6" gracePeriod=30
Sep 09 21:39:01.886629 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:01.886546    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:01.974080 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:01.974050    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:01.992067 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:01.992032    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:39:02.024305 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.024269    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-kubeconfig\") pod \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") "
Sep 09 21:39:02.024305 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.024298    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-token\") pod \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") "
Sep 09 21:39:02.024524 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.024319    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpbkj\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-kube-api-access-rpbkj\") pod \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") "
Sep 09 21:39:02.024524 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.024340    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-ca-configmap\") pod \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") "
Sep 09 21:39:02.024524 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.024363    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-dns-cert\") pod \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") "
Sep 09 21:39:02.024735 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.024711    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "79f5a126-f044-449f-b0e4-1c89b58d8c8b" (UID: "79f5a126-f044-449f-b0e4-1c89b58d8c8b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:39:02.024832 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.024808    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-cacerts\") pod \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") "
Sep 09 21:39:02.024896 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.024880    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f5a126-f044-449f-b0e4-1c89b58d8c8b-local-certs\") pod \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\" (UID: \"79f5a126-f044-449f-b0e4-1c89b58d8c8b\") "
Sep 09 21:39:02.025642 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.025617    2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-ca-configmap\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:39:02.027606 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.027531    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "79f5a126-f044-449f-b0e4-1c89b58d8c8b" (UID: "79f5a126-f044-449f-b0e4-1c89b58d8c8b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:39:02.027606 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.027544    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-kube-api-access-rpbkj" (OuterVolumeSpecName: "kube-api-access-rpbkj") pod "79f5a126-f044-449f-b0e4-1c89b58d8c8b" (UID: "79f5a126-f044-449f-b0e4-1c89b58d8c8b"). InnerVolumeSpecName "kube-api-access-rpbkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:39:02.027606 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.027552    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "79f5a126-f044-449f-b0e4-1c89b58d8c8b" (UID: "79f5a126-f044-449f-b0e4-1c89b58d8c8b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:39:02.027606 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.027565    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-token" (OuterVolumeSpecName: "istio-token") pod "79f5a126-f044-449f-b0e4-1c89b58d8c8b" (UID: "79f5a126-f044-449f-b0e4-1c89b58d8c8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:39:02.027983 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.027948    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f5a126-f044-449f-b0e4-1c89b58d8c8b-local-certs" (OuterVolumeSpecName: "local-certs") pod "79f5a126-f044-449f-b0e4-1c89b58d8c8b" (UID: "79f5a126-f044-449f-b0e4-1c89b58d8c8b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:02.028137 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.028122    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-cacerts" (OuterVolumeSpecName: "cacerts") pod "79f5a126-f044-449f-b0e4-1c89b58d8c8b" (UID: "79f5a126-f044-449f-b0e4-1c89b58d8c8b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:39:02.091157 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.091114    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:02.128648 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.127011    2585 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-cacerts\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:39:02.128648 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.127043    2585 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f5a126-f044-449f-b0e4-1c89b58d8c8b-local-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:39:02.128648 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.127058    2585 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-kubeconfig\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:39:02.128648 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.127077    2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-token\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:39:02.128648 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.127091    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbkj\" (UniqueName: \"kubernetes.io/projected/79f5a126-f044-449f-b0e4-1c89b58d8c8b-kube-api-access-rpbkj\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:39:02.128648 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.127104    2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f5a126-f044-449f-b0e4-1c89b58d8c8b-istio-csr-dns-cert\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:39:02.429035 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.428970    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:02.614484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.614447    2585 generic.go:356] "Generic (PLEG): container finished" podID="79f5a126-f044-449f-b0e4-1c89b58d8c8b" containerID="044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6" exitCode=0
Sep 09 21:39:02.614940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.614512    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"
Sep 09 21:39:02.614940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.614536    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6" event={"ID":"79f5a126-f044-449f-b0e4-1c89b58d8c8b","Type":"ContainerDied","Data":"044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6"}
Sep 09 21:39:02.614940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.614585    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6" event={"ID":"79f5a126-f044-449f-b0e4-1c89b58d8c8b","Type":"ContainerDied","Data":"d3e5f61d5c6955f27b5c9a71be858db2bcd4941752f2f87ef57cff8df6cb397a"}
Sep 09 21:39:02.615643 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.615625    2585 scope.go:117] "RemoveContainer" containerID="044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6"
Sep 09 21:39:02.625299 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.625269    2585 scope.go:117] "RemoveContainer" containerID="044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6"
Sep 09 21:39:02.625565 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:39:02.625544    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6\": container with ID starting with 044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6 not found: ID does not exist" containerID="044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6"
Sep 09 21:39:02.625617 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.625579    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6"} err="failed to get container status \"044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6\": rpc error: code = NotFound desc = could not find container \"044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6\": container with ID starting with 044cea792acd5356c42c7cdaa5969c7b2ecf91dabd107fbdace537828188dac6 not found: ID does not exist"
Sep 09 21:39:02.651009 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.650981    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"]
Sep 09 21:39:02.663044 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:02.663022    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xm4k6"]
Sep 09 21:39:03.173100 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:03.173066    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:03.343171 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:03.343140    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:04.126264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:04.126232    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f5a126-f044-449f-b0e4-1c89b58d8c8b" path="/var/lib/kubelet/pods/79f5a126-f044-449f-b0e4-1c89b58d8c8b/volumes"
Sep 09 21:39:10.267547 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:10.267465    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:10.325354 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:10.325328    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:39.697463 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:39.697431    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:39.797450 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:39.797420    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:51.015159 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:51.015123    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:51.115443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:51.115405    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:51.166313 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:51.166280    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:53.050598 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:53.050569    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:53.148397 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:39:53.148367    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:06.207080 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:06.207044    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:06.987846 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:06.987813    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:07.873198 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:07.873157    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:10.748722 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.748690    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"]
Sep 09 21:40:10.749218 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.748989    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f5a126-f044-449f-b0e4-1c89b58d8c8b" containerName="discovery"
Sep 09 21:40:10.749218 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.749002    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f5a126-f044-449f-b0e4-1c89b58d8c8b" containerName="discovery"
Sep 09 21:40:10.749218 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.749051    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f5a126-f044-449f-b0e4-1c89b58d8c8b" containerName="discovery"
Sep 09 21:40:10.751884 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.751869    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:10.754730 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.754710    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rx9gw\""
Sep 09 21:40:10.755259 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.755240    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 21:40:10.755342 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.755247    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 21:40:10.764839 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.764815    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"]
Sep 09 21:40:10.849309 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.849281    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0314a21-56ca-4fe3-90e0-5dc421c08223-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" (UID: \"b0314a21-56ca-4fe3-90e0-5dc421c08223\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:10.849482 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.849334    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcnqs\" (UniqueName: \"kubernetes.io/projected/b0314a21-56ca-4fe3-90e0-5dc421c08223-kube-api-access-wcnqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" (UID: \"b0314a21-56ca-4fe3-90e0-5dc421c08223\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:10.950633 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.950603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0314a21-56ca-4fe3-90e0-5dc421c08223-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" (UID: \"b0314a21-56ca-4fe3-90e0-5dc421c08223\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:10.950795 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.950667    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcnqs\" (UniqueName: \"kubernetes.io/projected/b0314a21-56ca-4fe3-90e0-5dc421c08223-kube-api-access-wcnqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" (UID: \"b0314a21-56ca-4fe3-90e0-5dc421c08223\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:10.951003 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.950982    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0314a21-56ca-4fe3-90e0-5dc421c08223-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" (UID: \"b0314a21-56ca-4fe3-90e0-5dc421c08223\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:10.960174 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:10.960150    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcnqs\" (UniqueName: \"kubernetes.io/projected/b0314a21-56ca-4fe3-90e0-5dc421c08223-kube-api-access-wcnqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" (UID: \"b0314a21-56ca-4fe3-90e0-5dc421c08223\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:11.061431 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:11.061350    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:11.339807 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:11.339784    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"]
Sep 09 21:40:11.341607 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:40:11.341576    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0314a21_56ca_4fe3_90e0_5dc421c08223.slice/crio-db24b3277f271cf1f7891a6ee620ab9e437e500d4905ff20189170df2048a279 WatchSource:0}: Error finding container db24b3277f271cf1f7891a6ee620ab9e437e500d4905ff20189170df2048a279: Status 404 returned error can't find the container with id db24b3277f271cf1f7891a6ee620ab9e437e500d4905ff20189170df2048a279
Sep 09 21:40:11.892461 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:11.892427    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks" event={"ID":"b0314a21-56ca-4fe3-90e0-5dc421c08223","Type":"ContainerStarted","Data":"db24b3277f271cf1f7891a6ee620ab9e437e500d4905ff20189170df2048a279"}
Sep 09 21:40:11.954987 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:11.954952    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:12.055405 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:12.055372    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:12.688285 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:12.688249    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"]
Sep 09 21:40:12.691623 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:12.691603    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:12.694022 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:12.693999    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6cpk6\""
Sep 09 21:40:12.701660 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:12.701632    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"]
Sep 09 21:40:12.768278 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:12.768243    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znc6\" (UniqueName: \"kubernetes.io/projected/d1accdb1-cd60-4cac-9ea1-e670a4d35851-kube-api-access-6znc6\") pod \"limitador-operator-controller-manager-74495c69f8-7vc44\" (UID: \"d1accdb1-cd60-4cac-9ea1-e670a4d35851\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:12.869141 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:12.869108    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znc6\" (UniqueName: \"kubernetes.io/projected/d1accdb1-cd60-4cac-9ea1-e670a4d35851-kube-api-access-6znc6\") pod \"limitador-operator-controller-manager-74495c69f8-7vc44\" (UID: \"d1accdb1-cd60-4cac-9ea1-e670a4d35851\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:12.882260 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:12.882229    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znc6\" (UniqueName: \"kubernetes.io/projected/d1accdb1-cd60-4cac-9ea1-e670a4d35851-kube-api-access-6znc6\") pod \"limitador-operator-controller-manager-74495c69f8-7vc44\" (UID: \"d1accdb1-cd60-4cac-9ea1-e670a4d35851\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:13.005902 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:13.005829    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:13.009998 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:13.009957    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:13.289844 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:13.289761    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"]
Sep 09 21:40:13.294519 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:40:13.294488    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1accdb1_cd60_4cac_9ea1_e670a4d35851.slice/crio-088aa5ef7332df90dc6e9e9dd18af55e60625eade2952389d55ca528493259e7 WatchSource:0}: Error finding container 088aa5ef7332df90dc6e9e9dd18af55e60625eade2952389d55ca528493259e7: Status 404 returned error can't find the container with id 088aa5ef7332df90dc6e9e9dd18af55e60625eade2952389d55ca528493259e7
Sep 09 21:40:13.390513 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:13.390482    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:13.904304 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:13.904270    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44" event={"ID":"d1accdb1-cd60-4cac-9ea1-e670a4d35851","Type":"ContainerStarted","Data":"088aa5ef7332df90dc6e9e9dd18af55e60625eade2952389d55ca528493259e7"}
Sep 09 21:40:14.044250 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:14.044178    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:15.400703 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:15.400671    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-42b8p"]
Sep 09 21:40:15.404804 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:15.404782    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-42b8p"
Sep 09 21:40:15.407268 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:15.407249    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tdfx6\""
Sep 09 21:40:15.414345 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:15.414323    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-42b8p"]
Sep 09 21:40:15.495849 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:15.495810    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsm6b\" (UniqueName: \"kubernetes.io/projected/6f6ba88e-9177-4b70-b6c5-f8039ed66887-kube-api-access-qsm6b\") pod \"authorino-operator-76b478bd78-42b8p\" (UID: \"6f6ba88e-9177-4b70-b6c5-f8039ed66887\") " pod="kuadrant-system/authorino-operator-76b478bd78-42b8p"
Sep 09 21:40:15.597170 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:15.597135    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsm6b\" (UniqueName: \"kubernetes.io/projected/6f6ba88e-9177-4b70-b6c5-f8039ed66887-kube-api-access-qsm6b\") pod \"authorino-operator-76b478bd78-42b8p\" (UID: \"6f6ba88e-9177-4b70-b6c5-f8039ed66887\") " pod="kuadrant-system/authorino-operator-76b478bd78-42b8p"
Sep 09 21:40:15.609115 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:15.606738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsm6b\" (UniqueName: \"kubernetes.io/projected/6f6ba88e-9177-4b70-b6c5-f8039ed66887-kube-api-access-qsm6b\") pod \"authorino-operator-76b478bd78-42b8p\" (UID: \"6f6ba88e-9177-4b70-b6c5-f8039ed66887\") " pod="kuadrant-system/authorino-operator-76b478bd78-42b8p"
Sep 09 21:40:15.718068 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:15.718032    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-42b8p"
Sep 09 21:40:16.038593 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:16.038490    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:16.251496 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:16.251454    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:17.103952 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:17.103914    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-42b8p"]
Sep 09 21:40:17.106514 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:40:17.106489    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6ba88e_9177_4b70_b6c5_f8039ed66887.slice/crio-d00a2ffe41aace9cb7aafe521f4728c4044253db37b6a53bdded655e479283e7 WatchSource:0}: Error finding container d00a2ffe41aace9cb7aafe521f4728c4044253db37b6a53bdded655e479283e7: Status 404 returned error can't find the container with id d00a2ffe41aace9cb7aafe521f4728c4044253db37b6a53bdded655e479283e7
Sep 09 21:40:17.641266 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:17.641230    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:17.924705 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:17.924669    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-42b8p" event={"ID":"6f6ba88e-9177-4b70-b6c5-f8039ed66887","Type":"ContainerStarted","Data":"d00a2ffe41aace9cb7aafe521f4728c4044253db37b6a53bdded655e479283e7"}
Sep 09 21:40:17.926913 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:17.926882    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks" event={"ID":"b0314a21-56ca-4fe3-90e0-5dc421c08223","Type":"ContainerStarted","Data":"38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c"}
Sep 09 21:40:17.927233 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:17.927147    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:17.929017 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:17.928905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44" event={"ID":"d1accdb1-cd60-4cac-9ea1-e670a4d35851","Type":"ContainerStarted","Data":"23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b"}
Sep 09 21:40:17.929017 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:17.928970    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:17.977975 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:17.977823    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks" podStartSLOduration=2.474417893 podStartE2EDuration="7.9778019s" totalImagesPullingTime="5.503384007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:40:17.974990774 +0000 UTC m=+420.349314598" watchObservedRunningTime="2026-09-09 21:40:17.9778019 +0000 UTC m=+420.352125725"
Sep 09 21:40:18.008356 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:18.008297    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44" podStartSLOduration=2.459530304 podStartE2EDuration="6.008275506s" totalImagesPullingTime="3.548745202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:40:18.00715538 +0000 UTC m=+420.381479213" watchObservedRunningTime="2026-09-09 21:40:18.008275506 +0000 UTC m=+420.382599331"
Sep 09 21:40:18.934921 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:18.934879    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-42b8p" event={"ID":"6f6ba88e-9177-4b70-b6c5-f8039ed66887","Type":"ContainerStarted","Data":"9456671304d86e3d294dbbc210ca74afc46d8b4193a6a33f793520f6597ff291"}
Sep 09 21:40:18.935335 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:18.935156    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-42b8p"
Sep 09 21:40:18.951678 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:18.951633    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-42b8p" podStartSLOduration=2.494781969 podStartE2EDuration="3.951619359s" totalImagesPullingTime="1.45683739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:40:18.949617744 +0000 UTC m=+421.323941568" watchObservedRunningTime="2026-09-09 21:40:18.951619359 +0000 UTC m=+421.325943182"
Sep 09 21:40:20.728675 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:20.728638    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:20.828075 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:20.828041    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:21.596173 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:21.596139    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:21.649029 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:21.648998    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:22.004801 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:22.004770    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:22.054303 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:22.054278    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:22.403842 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:22.403813    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:22.515928 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:22.515901    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:28.614374 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:28.614337    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:28.715633 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:28.715604    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:28.936718 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:28.936691    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:28.936877 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:28.936757    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:29.943337 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:29.943301    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-42b8p"
Sep 09 21:40:30.987014 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:30.986975    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"]
Sep 09 21:40:30.987508 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:30.987192    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" containerName="manager" containerID="cri-o://38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c" gracePeriod=2
Sep 09 21:40:30.998443 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:30.998417    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"]
Sep 09 21:40:31.002962 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.002928    2585 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b0314a21-56ca-4fe3-90e0-5dc421c08223\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-42kks\": pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.010538 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.010518    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"]
Sep 09 21:40:31.010807 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.010796    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" containerName="manager"
Sep 09 21:40:31.010854 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.010810    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" containerName="manager"
Sep 09 21:40:31.010888 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.010876    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" containerName="manager"
Sep 09 21:40:31.013777 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.013759    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"]
Sep 09 21:40:31.013932 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.013907    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:31.013932 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.013916    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" containerName="manager" containerID="cri-o://23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b" gracePeriod=2
Sep 09 21:40:31.015857 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.015830    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.024232 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.024180    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"]
Sep 09 21:40:31.024541 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.024510    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5630133-ffa9-4b35-9ae6-6d15e0330f23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nkv9s\" (UID: \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:31.024651 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.024597    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ftds\" (UniqueName: \"kubernetes.io/projected/b5630133-ffa9-4b35-9ae6-6d15e0330f23-kube-api-access-2ftds\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nkv9s\" (UID: \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:31.031493 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.031470    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"]
Sep 09 21:40:31.033627 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.033600    2585 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d1accdb1-cd60-4cac-9ea1-e670a4d35851\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-7vc44\": pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:31.035318 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.035294    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:31.036960 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.036941    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.043299 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.043278    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"]
Sep 09 21:40:31.043558 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.043547    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" containerName="manager"
Sep 09 21:40:31.043609 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.043560    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" containerName="manager"
Sep 09 21:40:31.043649 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.043614    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" containerName="manager"
Sep 09 21:40:31.046482 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.046466    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"
Sep 09 21:40:31.058446 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.058413    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.061120 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.061095    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:31.062004 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.061983    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"]
Sep 09 21:40:31.125442 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.125414    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ftds\" (UniqueName: \"kubernetes.io/projected/b5630133-ffa9-4b35-9ae6-6d15e0330f23-kube-api-access-2ftds\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nkv9s\" (UID: \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:31.125558 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.125469    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j9rj\" (UniqueName: \"kubernetes.io/projected/a4ebf895-8937-4ca0-9144-66c3303a7e39-kube-api-access-5j9rj\") pod \"limitador-operator-controller-manager-74495c69f8-mjskk\" (UID: \"a4ebf895-8937-4ca0-9144-66c3303a7e39\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"
Sep 09 21:40:31.125558 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.125520    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5630133-ffa9-4b35-9ae6-6d15e0330f23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nkv9s\" (UID: \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:31.125931 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.125911    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5630133-ffa9-4b35-9ae6-6d15e0330f23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nkv9s\" (UID: \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:31.137112 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.137066    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ftds\" (UniqueName: \"kubernetes.io/projected/b5630133-ffa9-4b35-9ae6-6d15e0330f23-kube-api-access-2ftds\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nkv9s\" (UID: \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:31.146653 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.146633    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:31.226639 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.226580    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j9rj\" (UniqueName: \"kubernetes.io/projected/a4ebf895-8937-4ca0-9144-66c3303a7e39-kube-api-access-5j9rj\") pod \"limitador-operator-controller-manager-74495c69f8-mjskk\" (UID: \"a4ebf895-8937-4ca0-9144-66c3303a7e39\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"
Sep 09 21:40:31.242345 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.242283    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j9rj\" (UniqueName: \"kubernetes.io/projected/a4ebf895-8937-4ca0-9144-66c3303a7e39-kube-api-access-5j9rj\") pod \"limitador-operator-controller-manager-74495c69f8-mjskk\" (UID: \"a4ebf895-8937-4ca0-9144-66c3303a7e39\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"
Sep 09 21:40:31.250076 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.250048    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"]
Sep 09 21:40:31.254797 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.254774    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:31.255911 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.255894    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.256694 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.256667    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.258395 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.258370    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:31.268997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.268978    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:31.272658 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.272629    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"]
Sep 09 21:40:31.301857 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.299667    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:31.301857 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.301299    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.327397 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.327373    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6znc6\" (UniqueName: \"kubernetes.io/projected/d1accdb1-cd60-4cac-9ea1-e670a4d35851-kube-api-access-6znc6\") pod \"d1accdb1-cd60-4cac-9ea1-e670a4d35851\" (UID: \"d1accdb1-cd60-4cac-9ea1-e670a4d35851\") "
Sep 09 21:40:31.327505 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.327418    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcnqs\" (UniqueName: \"kubernetes.io/projected/b0314a21-56ca-4fe3-90e0-5dc421c08223-kube-api-access-wcnqs\") pod \"b0314a21-56ca-4fe3-90e0-5dc421c08223\" (UID: \"b0314a21-56ca-4fe3-90e0-5dc421c08223\") "
Sep 09 21:40:31.327505 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.327453    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0314a21-56ca-4fe3-90e0-5dc421c08223-extensions-socket-volume\") pod \"b0314a21-56ca-4fe3-90e0-5dc421c08223\" (UID: \"b0314a21-56ca-4fe3-90e0-5dc421c08223\") "
Sep 09 21:40:31.327580 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.327544    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvw4\" (UniqueName: \"kubernetes.io/projected/90e25932-0d31-4c53-944a-4a3dd2093b20-kube-api-access-vjvw4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8mp2k\" (UID: \"90e25932-0d31-4c53-944a-4a3dd2093b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:31.327580 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.327565    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90e25932-0d31-4c53-944a-4a3dd2093b20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8mp2k\" (UID: \"90e25932-0d31-4c53-944a-4a3dd2093b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:31.327894 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.327870    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0314a21-56ca-4fe3-90e0-5dc421c08223-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b0314a21-56ca-4fe3-90e0-5dc421c08223" (UID: "b0314a21-56ca-4fe3-90e0-5dc421c08223"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:40:31.329655 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.329633    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1accdb1-cd60-4cac-9ea1-e670a4d35851-kube-api-access-6znc6" (OuterVolumeSpecName: "kube-api-access-6znc6") pod "d1accdb1-cd60-4cac-9ea1-e670a4d35851" (UID: "d1accdb1-cd60-4cac-9ea1-e670a4d35851"). InnerVolumeSpecName "kube-api-access-6znc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:40:31.329655 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.329642    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0314a21-56ca-4fe3-90e0-5dc421c08223-kube-api-access-wcnqs" (OuterVolumeSpecName: "kube-api-access-wcnqs") pod "b0314a21-56ca-4fe3-90e0-5dc421c08223" (UID: "b0314a21-56ca-4fe3-90e0-5dc421c08223"). InnerVolumeSpecName "kube-api-access-wcnqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:40:31.428356 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.428322    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvw4\" (UniqueName: \"kubernetes.io/projected/90e25932-0d31-4c53-944a-4a3dd2093b20-kube-api-access-vjvw4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8mp2k\" (UID: \"90e25932-0d31-4c53-944a-4a3dd2093b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:31.428356 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.428360    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90e25932-0d31-4c53-944a-4a3dd2093b20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8mp2k\" (UID: \"90e25932-0d31-4c53-944a-4a3dd2093b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:31.428561 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.428439    2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0314a21-56ca-4fe3-90e0-5dc421c08223-extensions-socket-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:40:31.428561 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.428450    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6znc6\" (UniqueName: \"kubernetes.io/projected/d1accdb1-cd60-4cac-9ea1-e670a4d35851-kube-api-access-6znc6\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:40:31.428561 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.428462    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcnqs\" (UniqueName: \"kubernetes.io/projected/b0314a21-56ca-4fe3-90e0-5dc421c08223-kube-api-access-wcnqs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:40:31.428768 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.428750    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90e25932-0d31-4c53-944a-4a3dd2093b20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8mp2k\" (UID: \"90e25932-0d31-4c53-944a-4a3dd2093b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:31.448178 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.448144    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvw4\" (UniqueName: \"kubernetes.io/projected/90e25932-0d31-4c53-944a-4a3dd2093b20-kube-api-access-vjvw4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8mp2k\" (UID: \"90e25932-0d31-4c53-944a-4a3dd2093b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:31.453549 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.453518    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"
Sep 09 21:40:31.455610 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.455589    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"]
Sep 09 21:40:31.458826 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:40:31.458803    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5630133_ffa9_4b35_9ae6_6d15e0330f23.slice/crio-c3c8454d58a9fdc0d841e101851ef085ab43c5020a124791124dd578aef91539 WatchSource:0}: Error finding container c3c8454d58a9fdc0d841e101851ef085ab43c5020a124791124dd578aef91539: Status 404 returned error can't find the container with id c3c8454d58a9fdc0d841e101851ef085ab43c5020a124791124dd578aef91539
Sep 09 21:40:31.564817 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.564787    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:31.734396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.734372    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"]
Sep 09 21:40:31.737033 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:40:31.737003    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ebf895_8937_4ca0_9144_66c3303a7e39.slice/crio-197a613851018230d1bef86fe3b4f071e09d911d56e1c53f8a482d03c99b0e33 WatchSource:0}: Error finding container 197a613851018230d1bef86fe3b4f071e09d911d56e1c53f8a482d03c99b0e33: Status 404 returned error can't find the container with id 197a613851018230d1bef86fe3b4f071e09d911d56e1c53f8a482d03c99b0e33
Sep 09 21:40:31.845590 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.845556    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"]
Sep 09 21:40:31.849178 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:40:31.849156    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e25932_0d31_4c53_944a_4a3dd2093b20.slice/crio-4a6e652a499781be1d9f89883de77d9a8b0e3abf00ff54a628634d8fe49b6140 WatchSource:0}: Error finding container 4a6e652a499781be1d9f89883de77d9a8b0e3abf00ff54a628634d8fe49b6140: Status 404 returned error can't find the container with id 4a6e652a499781be1d9f89883de77d9a8b0e3abf00ff54a628634d8fe49b6140
Sep 09 21:40:31.996308 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.996274    2585 generic.go:356] "Generic (PLEG): container finished" podID="b0314a21-56ca-4fe3-90e0-5dc421c08223" containerID="38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c" exitCode=0
Sep 09 21:40:31.996727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.996333    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.996727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.996357    2585 scope.go:117] "RemoveContainer" containerID="38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c"
Sep 09 21:40:31.998602 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.998555    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k" event={"ID":"90e25932-0d31-4c53-944a-4a3dd2093b20","Type":"ContainerStarted","Data":"50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5"}
Sep 09 21:40:31.998709 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.998617    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k" event={"ID":"90e25932-0d31-4c53-944a-4a3dd2093b20","Type":"ContainerStarted","Data":"4a6e652a499781be1d9f89883de77d9a8b0e3abf00ff54a628634d8fe49b6140"}
Sep 09 21:40:31.998772 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:31.998709    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:31.998772 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:31.998735    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:32.000397 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.000337    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:32.000727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.000701    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk" event={"ID":"a4ebf895-8937-4ca0-9144-66c3303a7e39","Type":"ContainerStarted","Data":"06ed33df476cf6b7d7d9466aed3d03af3c027da93edc3a152f1af4f66ddf01f2"}
Sep 09 21:40:32.000839 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.000757    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"
Sep 09 21:40:32.000839 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.000781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk" event={"ID":"a4ebf895-8937-4ca0-9144-66c3303a7e39","Type":"ContainerStarted","Data":"197a613851018230d1bef86fe3b4f071e09d911d56e1c53f8a482d03c99b0e33"}
Sep 09 21:40:32.001962 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.001937    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:32.002072 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.002054    2585 generic.go:356] "Generic (PLEG): container finished" podID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" containerID="23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b" exitCode=0
Sep 09 21:40:32.002133 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.002105    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:32.003539 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.003512    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:32.003894 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.003874    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s" event={"ID":"b5630133-ffa9-4b35-9ae6-6d15e0330f23","Type":"ContainerStarted","Data":"c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754"}
Sep 09 21:40:32.003966 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.003911    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s" event={"ID":"b5630133-ffa9-4b35-9ae6-6d15e0330f23","Type":"ContainerStarted","Data":"c3c8454d58a9fdc0d841e101851ef085ab43c5020a124791124dd578aef91539"}
Sep 09 21:40:32.004007 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.003998    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:32.024274 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.024236    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:32.024946 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.024899    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k" podStartSLOduration=1.024878621 podStartE2EDuration="1.024878621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:40:32.022191881 +0000 UTC m=+434.396515705" watchObservedRunningTime="2026-09-09 21:40:32.024878621 +0000 UTC m=+434.399202449"
Sep 09 21:40:32.031335 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.031287    2585 scope.go:117] "RemoveContainer" containerID="38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c"
Sep 09 21:40:32.032726 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.032704    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c\": container with ID starting with 38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c not found: ID does not exist" containerID="38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c"
Sep 09 21:40:32.032806 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.032735    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c"} err="failed to get container status \"38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c\": rpc error: code = NotFound desc = could not find container \"38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c\": container with ID starting with 38d50a9e997852715a73c8f867672755396355c4511bc132e0afa702a0147c9c not found: ID does not exist"
Sep 09 21:40:32.032806 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.032752    2585 scope.go:117] "RemoveContainer" containerID="23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b"
Sep 09 21:40:32.041322 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.041302    2585 scope.go:117] "RemoveContainer" containerID="23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b"
Sep 09 21:40:32.041593 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.041577    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b\": container with ID starting with 23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b not found: ID does not exist" containerID="23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b"
Sep 09 21:40:32.041652 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.041599    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b"} err="failed to get container status \"23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b\": rpc error: code = NotFound desc = could not find container \"23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b\": container with ID starting with 23c4a76be0a883cf79279e367c4bb94315b9e58978c76b45ae03e91245cfb77b not found: ID does not exist"
Sep 09 21:40:32.045953 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.045905    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s" podStartSLOduration=2.045891796 podStartE2EDuration="2.045891796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:40:32.045357133 +0000 UTC m=+434.419680959" watchObservedRunningTime="2026-09-09 21:40:32.045891796 +0000 UTC m=+434.420215619"
Sep 09 21:40:32.047105 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.047082    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:32.072776 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.072731    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk" podStartSLOduration=1.072717941 podStartE2EDuration="1.072717941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:40:32.071470644 +0000 UTC m=+434.445794469" watchObservedRunningTime="2026-09-09 21:40:32.072717941 +0000 UTC m=+434.447041764"
Sep 09 21:40:32.073257 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.073180    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7vc44\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7vc44"
Sep 09 21:40:32.074858 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:32.074839    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-42kks\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42kks"
Sep 09 21:40:32.127936 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.127906    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0314a21-56ca-4fe3-90e0-5dc421c08223" path="/var/lib/kubelet/pods/b0314a21-56ca-4fe3-90e0-5dc421c08223/volumes"
Sep 09 21:40:32.128301 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.128287    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1accdb1-cd60-4cac-9ea1-e670a4d35851" path="/var/lib/kubelet/pods/d1accdb1-cd60-4cac-9ea1-e670a4d35851/volumes"
Sep 09 21:40:32.636773 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:32.636741    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:33.435567 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:33.435534    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:33.530882 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:33.530846    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:35.850482 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:35.850444    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:35.902934 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:35.902901    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:42.817378 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:42.817305    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:42.917613 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:42.917587    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:43.012039 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.012010    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:43.012221 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.012085    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:40:43.012221 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.012126    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjskk"
Sep 09 21:40:43.082600 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.082517    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"]
Sep 09 21:40:43.082766 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.082736    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s" podUID="b5630133-ffa9-4b35-9ae6-6d15e0330f23" containerName="manager" containerID="cri-o://c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754" gracePeriod=10
Sep 09 21:40:43.231772 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.231715    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:43.336957 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.336884    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:43.366807 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.366775    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:43.427147 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.427118    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5630133-ffa9-4b35-9ae6-6d15e0330f23-extensions-socket-volume\") pod \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\" (UID: \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\") "
Sep 09 21:40:43.427344 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.427183    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ftds\" (UniqueName: \"kubernetes.io/projected/b5630133-ffa9-4b35-9ae6-6d15e0330f23-kube-api-access-2ftds\") pod \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\" (UID: \"b5630133-ffa9-4b35-9ae6-6d15e0330f23\") "
Sep 09 21:40:43.427552 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.427529    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5630133-ffa9-4b35-9ae6-6d15e0330f23-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b5630133-ffa9-4b35-9ae6-6d15e0330f23" (UID: "b5630133-ffa9-4b35-9ae6-6d15e0330f23"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:40:43.429349 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.429325    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5630133-ffa9-4b35-9ae6-6d15e0330f23-kube-api-access-2ftds" (OuterVolumeSpecName: "kube-api-access-2ftds") pod "b5630133-ffa9-4b35-9ae6-6d15e0330f23" (UID: "b5630133-ffa9-4b35-9ae6-6d15e0330f23"). InnerVolumeSpecName "kube-api-access-2ftds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:40:43.445268 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.445240    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:43.528640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.528613    2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5630133-ffa9-4b35-9ae6-6d15e0330f23-extensions-socket-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:40:43.528640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.528636    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ftds\" (UniqueName: \"kubernetes.io/projected/b5630133-ffa9-4b35-9ae6-6d15e0330f23-kube-api-access-2ftds\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:40:43.547313 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:43.547284    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:44.067041 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.067001    2585 generic.go:356] "Generic (PLEG): container finished" podID="b5630133-ffa9-4b35-9ae6-6d15e0330f23" containerID="c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754" exitCode=0
Sep 09 21:40:44.067041 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.067047    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"
Sep 09 21:40:44.067531 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.067083    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s" event={"ID":"b5630133-ffa9-4b35-9ae6-6d15e0330f23","Type":"ContainerDied","Data":"c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754"}
Sep 09 21:40:44.067531 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.067133    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s" event={"ID":"b5630133-ffa9-4b35-9ae6-6d15e0330f23","Type":"ContainerDied","Data":"c3c8454d58a9fdc0d841e101851ef085ab43c5020a124791124dd578aef91539"}
Sep 09 21:40:44.067531 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.067157    2585 scope.go:117] "RemoveContainer" containerID="c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754"
Sep 09 21:40:44.075645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.075629    2585 scope.go:117] "RemoveContainer" containerID="c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754"
Sep 09 21:40:44.075920 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:40:44.075900    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754\": container with ID starting with c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754 not found: ID does not exist" containerID="c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754"
Sep 09 21:40:44.075964 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.075929    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754"} err="failed to get container status \"c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754\": rpc error: code = NotFound desc = could not find container \"c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754\": container with ID starting with c68dab45675798aa1b70c92841c8d0a069d9983b1052897278bf9e7c004ce754 not found: ID does not exist"
Sep 09 21:40:44.091697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.091676    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"]
Sep 09 21:40:44.096767 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.096748    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nkv9s"]
Sep 09 21:40:44.130143 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:44.130113    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5630133-ffa9-4b35-9ae6-6d15e0330f23" path="/var/lib/kubelet/pods/b5630133-ffa9-4b35-9ae6-6d15e0330f23/volumes"
Sep 09 21:40:45.980334 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:45.980298    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:50.437491 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:50.437450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:50.544681 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:50.544649    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:55.240753 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:55.240716    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:55.340084 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:55.340056    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:57.739116 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:57.739077    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:57.839183 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:57.839137    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:59.344246 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:59.344181    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:59.490618 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:40:59.490581    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:01.165606 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:01.165575    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:01.263387 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:01.263353    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:01.323124 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:01.323067    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:01.976262 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:01.976227    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:02.056836 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:02.056803    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:17.227441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:17.227410    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:17.326682 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:17.326626    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:17.730101 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:17.730064    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:29.651171 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:29.651134    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:29.751097 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:29.751044    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:32.222266 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:32.222237    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:36.010528 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.010487    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:36.040465 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.040434    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5654c46f85-smtdg"]
Sep 09 21:41:36.040734 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.040721    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5630133-ffa9-4b35-9ae6-6d15e0330f23" containerName="manager"
Sep 09 21:41:36.040734 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.040734    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5630133-ffa9-4b35-9ae6-6d15e0330f23" containerName="manager"
Sep 09 21:41:36.040966 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.040926    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5630133-ffa9-4b35-9ae6-6d15e0330f23" containerName="manager"
Sep 09 21:41:36.045309 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.045289    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.047452 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.047431    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 21:41:36.047560 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.047481    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dqzb\""
Sep 09 21:41:36.047560 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.047431    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 21:41:36.047670 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.047457    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 21:41:36.054773 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.054750    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5654c46f85-smtdg"]
Sep 09 21:41:36.078943 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.078920    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:36.133370 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.133341    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:36.174422 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.174388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfb72\" (UniqueName: \"kubernetes.io/projected/cfaa6def-d131-4cae-8704-279232804b49-kube-api-access-tfb72\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.174595 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.174447    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-metrics-certs\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.174595 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.174566    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-webhook-cert\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.192559 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.192534    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4f964485-rr8r5"]
Sep 09 21:41:36.194861 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.194827    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:36.195653 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.195638    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.214152 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.214127    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4f964485-rr8r5"]
Sep 09 21:41:36.252575 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.252537    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:36.275196 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.275134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-metrics-certs\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.275196 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.275182    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-webhook-cert\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.275363 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.275224    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-metrics-certs\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.275363 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.275255    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-webhook-cert\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.275441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.275390    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfb72\" (UniqueName: \"kubernetes.io/projected/cfaa6def-d131-4cae-8704-279232804b49-kube-api-access-tfb72\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.275441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.275425    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsjzn\" (UniqueName: \"kubernetes.io/projected/480d0adf-08ef-4d0c-a495-c9d1ecb54532-kube-api-access-rsjzn\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.277682 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.277662    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-metrics-certs\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.277774 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.277734    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-webhook-cert\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.283016 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.282992    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfb72\" (UniqueName: \"kubernetes.io/projected/cfaa6def-d131-4cae-8704-279232804b49-kube-api-access-tfb72\") pod \"maas-controller-5654c46f85-smtdg\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") " pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.357159 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.357130    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:36.376472 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.376439    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-webhook-cert\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.376594 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.376495    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-metrics-certs\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.376594 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.376579    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsjzn\" (UniqueName: \"kubernetes.io/projected/480d0adf-08ef-4d0c-a495-c9d1ecb54532-kube-api-access-rsjzn\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.378962 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.378943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-metrics-certs\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.379152 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.379130    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-webhook-cert\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.384291 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.384266    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsjzn\" (UniqueName: \"kubernetes.io/projected/480d0adf-08ef-4d0c-a495-c9d1ecb54532-kube-api-access-rsjzn\") pod \"maas-controller-c4f964485-rr8r5\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") " pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.507049 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.507021    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:36.628933 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.628872    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5654c46f85-smtdg"]
Sep 09 21:41:36.631679 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:41:36.631654    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaa6def_d131_4cae_8704_279232804b49.slice/crio-66032b9c796418c19fe2d9a5fc4c4dec66e275589de9ff37e10b8c01bb81b36b WatchSource:0}: Error finding container 66032b9c796418c19fe2d9a5fc4c4dec66e275589de9ff37e10b8c01bb81b36b: Status 404 returned error can't find the container with id 66032b9c796418c19fe2d9a5fc4c4dec66e275589de9ff37e10b8c01bb81b36b
Sep 09 21:41:36.775301 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:36.775277    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4f964485-rr8r5"]
Sep 09 21:41:36.777476 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:41:36.777445    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480d0adf_08ef_4d0c_a495_c9d1ecb54532.slice/crio-564f70c979e426ee4941873c919f222ca27dd09da41a6f877eefb0fbd3385782 WatchSource:0}: Error finding container 564f70c979e426ee4941873c919f222ca27dd09da41a6f877eefb0fbd3385782: Status 404 returned error can't find the container with id 564f70c979e426ee4941873c919f222ca27dd09da41a6f877eefb0fbd3385782
Sep 09 21:41:37.296470 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:37.296429    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-rr8r5" event={"ID":"480d0adf-08ef-4d0c-a495-c9d1ecb54532","Type":"ContainerStarted","Data":"564f70c979e426ee4941873c919f222ca27dd09da41a6f877eefb0fbd3385782"}
Sep 09 21:41:37.297895 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:37.297870    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5654c46f85-smtdg" event={"ID":"cfaa6def-d131-4cae-8704-279232804b49","Type":"ContainerStarted","Data":"66032b9c796418c19fe2d9a5fc4c4dec66e275589de9ff37e10b8c01bb81b36b"}
Sep 09 21:41:39.495366 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:39.495334    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:39.593503 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:39.593418    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:40.315388 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:40.315354    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-rr8r5" event={"ID":"480d0adf-08ef-4d0c-a495-c9d1ecb54532","Type":"ContainerStarted","Data":"05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81"}
Sep 09 21:41:40.315572 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:40.315456    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:40.317061 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:40.317039    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5654c46f85-smtdg" event={"ID":"cfaa6def-d131-4cae-8704-279232804b49","Type":"ContainerStarted","Data":"c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df"}
Sep 09 21:41:40.317183 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:40.317077    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:40.354231 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:40.351007    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c4f964485-rr8r5" podStartSLOduration=1.794564912 podStartE2EDuration="4.350987861s" totalImagesPullingTime="2.556422949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:40.347405481 +0000 UTC m=+502.721729306" watchObservedRunningTime="2026-09-09 21:41:40.350987861 +0000 UTC m=+502.725311689"
Sep 09 21:41:40.379930 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:40.379873    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5654c46f85-smtdg" podStartSLOduration=1.6869965759999999 podStartE2EDuration="4.379855809s" totalImagesPullingTime="2.692859233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:40.378066842 +0000 UTC m=+502.752390668" watchObservedRunningTime="2026-09-09 21:41:40.379855809 +0000 UTC m=+502.754179631"
Sep 09 21:41:51.324354 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.324321    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 21:41:51.324825 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.324402    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:51.378687 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.378644    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5654c46f85-smtdg"]
Sep 09 21:41:51.378887 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.378856    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5654c46f85-smtdg" podUID="cfaa6def-d131-4cae-8704-279232804b49" containerName="manager" containerID="cri-o://c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df" gracePeriod=30
Sep 09 21:41:51.621630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.621608    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:51.703839 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.703806    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-metrics-certs\") pod \"cfaa6def-d131-4cae-8704-279232804b49\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") "
Sep 09 21:41:51.703979 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.703845    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfb72\" (UniqueName: \"kubernetes.io/projected/cfaa6def-d131-4cae-8704-279232804b49-kube-api-access-tfb72\") pod \"cfaa6def-d131-4cae-8704-279232804b49\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") "
Sep 09 21:41:51.704019 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.703979    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-webhook-cert\") pod \"cfaa6def-d131-4cae-8704-279232804b49\" (UID: \"cfaa6def-d131-4cae-8704-279232804b49\") "
Sep 09 21:41:51.706115 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.706081    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cfaa6def-d131-4cae-8704-279232804b49" (UID: "cfaa6def-d131-4cae-8704-279232804b49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:51.706115 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.706097    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cfaa6def-d131-4cae-8704-279232804b49" (UID: "cfaa6def-d131-4cae-8704-279232804b49"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:51.706288 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.706181    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaa6def-d131-4cae-8704-279232804b49-kube-api-access-tfb72" (OuterVolumeSpecName: "kube-api-access-tfb72") pod "cfaa6def-d131-4cae-8704-279232804b49" (UID: "cfaa6def-d131-4cae-8704-279232804b49"). InnerVolumeSpecName "kube-api-access-tfb72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:51.805651 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.805618    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:41:51.805651 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.805645    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfb72\" (UniqueName: \"kubernetes.io/projected/cfaa6def-d131-4cae-8704-279232804b49-kube-api-access-tfb72\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:41:51.805816 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:51.805658    2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfaa6def-d131-4cae-8704-279232804b49-webhook-cert\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:41:52.376057 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.376019    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfaa6def-d131-4cae-8704-279232804b49" containerID="c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df" exitCode=0
Sep 09 21:41:52.376529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.376072    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5654c46f85-smtdg"
Sep 09 21:41:52.376529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.376102    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5654c46f85-smtdg" event={"ID":"cfaa6def-d131-4cae-8704-279232804b49","Type":"ContainerDied","Data":"c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df"}
Sep 09 21:41:52.376529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.376170    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5654c46f85-smtdg" event={"ID":"cfaa6def-d131-4cae-8704-279232804b49","Type":"ContainerDied","Data":"66032b9c796418c19fe2d9a5fc4c4dec66e275589de9ff37e10b8c01bb81b36b"}
Sep 09 21:41:52.376529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.376191    2585 scope.go:117] "RemoveContainer" containerID="c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df"
Sep 09 21:41:52.384709 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.384477    2585 scope.go:117] "RemoveContainer" containerID="c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df"
Sep 09 21:41:52.384791 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:41:52.384772    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df\": container with ID starting with c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df not found: ID does not exist" containerID="c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df"
Sep 09 21:41:52.384838 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.384802    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df"} err="failed to get container status \"c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df\": rpc error: code = NotFound desc = could not find container \"c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df\": container with ID starting with c600c178332ceda724b09dab66eb0c9f3ff99c67336e813dc44abaccab0516df not found: ID does not exist"
Sep 09 21:41:52.393576 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.393553    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5654c46f85-smtdg"]
Sep 09 21:41:52.396801 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:52.396777    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5654c46f85-smtdg"]
Sep 09 21:41:54.126999 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:41:54.126966    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfaa6def-d131-4cae-8704-279232804b49" path="/var/lib/kubelet/pods/cfaa6def-d131-4cae-8704-279232804b49/volumes"
Sep 09 21:42:10.190108 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:10.190029    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:10.290488 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:10.290455    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:11.510720 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.510685    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:11.611525 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.611488    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:11.712372 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.712339    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:11.813095 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.813015    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:11.823638 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.823609    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"]
Sep 09 21:42:11.823959 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.823947    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfaa6def-d131-4cae-8704-279232804b49" containerName="manager"
Sep 09 21:42:11.824001 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.823962    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfaa6def-d131-4cae-8704-279232804b49" containerName="manager"
Sep 09 21:42:11.824035 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.824018    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfaa6def-d131-4cae-8704-279232804b49" containerName="manager"
Sep 09 21:42:11.825818 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.825803    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.829301 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.829281    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:42:11.829432 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.829394    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:42:11.829432 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.829411    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 21:42:11.829432 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.829394    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qxw5s\""
Sep 09 21:42:11.829575 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.829397    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 21:42:11.837799 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.837776    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"]
Sep 09 21:42:11.874716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.874694    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7356283-c0da-491b-879f-4db48a985a01-openshift-service-ca\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.874857 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.874732    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-metrics-certs\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.874857 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.874812    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-maas-api-tls\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.874857 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.874835    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj84m\" (UniqueName: \"kubernetes.io/projected/b7356283-c0da-491b-879f-4db48a985a01-kube-api-access-jj84m\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.976108 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.976077    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-maas-api-tls\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.976304 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.976118    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj84m\" (UniqueName: \"kubernetes.io/projected/b7356283-c0da-491b-879f-4db48a985a01-kube-api-access-jj84m\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.976304 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.976196    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7356283-c0da-491b-879f-4db48a985a01-openshift-service-ca\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.976304 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.976263    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-metrics-certs\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.976929 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.976901    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7356283-c0da-491b-879f-4db48a985a01-openshift-service-ca\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.978822 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.978790    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-maas-api-tls\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.978822 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.978805    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-metrics-certs\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:11.983345 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:11.983326    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj84m\" (UniqueName: \"kubernetes.io/projected/b7356283-c0da-491b-879f-4db48a985a01-kube-api-access-jj84m\") pod \"maas-api-55c59f7b5b-88sjk\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") " pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:12.057165 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.057117    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:12.136056 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.135988    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:12.150831 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.150799    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:12.151383 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.151362    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"]
Sep 09 21:42:12.153816 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.153793    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:12.156170 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.156151    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 21:42:12.156287 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.156198    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vd9p9\""
Sep 09 21:42:12.164513 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.164483    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"]
Sep 09 21:42:12.179826 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.179794    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2990b2d-f023-4113-a7b2-998167d03da9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6wpd7\" (UID: \"b2990b2d-f023-4113-a7b2-998167d03da9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:12.180082 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.180057    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6sq\" (UniqueName: \"kubernetes.io/projected/b2990b2d-f023-4113-a7b2-998167d03da9-kube-api-access-vj6sq\") pod \"payload-pre-processing-59fbc47bf8-6wpd7\" (UID: \"b2990b2d-f023-4113-a7b2-998167d03da9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:12.280985 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.280945    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6sq\" (UniqueName: \"kubernetes.io/projected/b2990b2d-f023-4113-a7b2-998167d03da9-kube-api-access-vj6sq\") pod \"payload-pre-processing-59fbc47bf8-6wpd7\" (UID: \"b2990b2d-f023-4113-a7b2-998167d03da9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:12.281162 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.281051    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2990b2d-f023-4113-a7b2-998167d03da9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6wpd7\" (UID: \"b2990b2d-f023-4113-a7b2-998167d03da9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:12.281775 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.281753    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2990b2d-f023-4113-a7b2-998167d03da9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6wpd7\" (UID: \"b2990b2d-f023-4113-a7b2-998167d03da9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:12.288860 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.288835    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6sq\" (UniqueName: \"kubernetes.io/projected/b2990b2d-f023-4113-a7b2-998167d03da9-kube-api-access-vj6sq\") pod \"payload-pre-processing-59fbc47bf8-6wpd7\" (UID: \"b2990b2d-f023-4113-a7b2-998167d03da9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:12.414227 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.414183    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"]
Sep 09 21:42:12.417380 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:42:12.417354    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7356283_c0da_491b_879f_4db48a985a01.slice/crio-c2779d0c7aec89f1c82b3b83a33564533944ef58af996d4b8ff9a375a363d100 WatchSource:0}: Error finding container c2779d0c7aec89f1c82b3b83a33564533944ef58af996d4b8ff9a375a363d100: Status 404 returned error can't find the container with id c2779d0c7aec89f1c82b3b83a33564533944ef58af996d4b8ff9a375a363d100
Sep 09 21:42:12.461345 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.461316    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk" event={"ID":"b7356283-c0da-491b-879f-4db48a985a01","Type":"ContainerStarted","Data":"c2779d0c7aec89f1c82b3b83a33564533944ef58af996d4b8ff9a375a363d100"}
Sep 09 21:42:12.468623 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.468601    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:12.741360 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:12.741293    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"]
Sep 09 21:42:12.744349 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:42:12.744320    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2990b2d_f023_4113_a7b2_998167d03da9.slice/crio-afde8dda20ccac4c1e0b5fd2d5ddbb59d4fc56feead4877caf15d14262d4585a WatchSource:0}: Error finding container afde8dda20ccac4c1e0b5fd2d5ddbb59d4fc56feead4877caf15d14262d4585a: Status 404 returned error can't find the container with id afde8dda20ccac4c1e0b5fd2d5ddbb59d4fc56feead4877caf15d14262d4585a
Sep 09 21:42:13.467579 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:13.467538    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7" event={"ID":"b2990b2d-f023-4113-a7b2-998167d03da9","Type":"ContainerStarted","Data":"afde8dda20ccac4c1e0b5fd2d5ddbb59d4fc56feead4877caf15d14262d4585a"}
Sep 09 21:42:15.477335 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:15.477292    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7" event={"ID":"b2990b2d-f023-4113-a7b2-998167d03da9","Type":"ContainerStarted","Data":"690629dbd0174dd0fb0a8f091279e5657bbb81df2f4defe66ec2126ee13cc771"}
Sep 09 21:42:15.477760 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:15.477357    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:15.478848 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:15.478814    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk" event={"ID":"b7356283-c0da-491b-879f-4db48a985a01","Type":"ContainerStarted","Data":"5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1"}
Sep 09 21:42:15.478983 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:15.478887    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:15.492538 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:15.492493    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7" podStartSLOduration=0.877456482 podStartE2EDuration="3.492479129s" totalImagesPullingTime="2.615022647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:15.491481476 +0000 UTC m=+537.865805299" watchObservedRunningTime="2026-09-09 21:42:15.492479129 +0000 UTC m=+537.866802953"
Sep 09 21:42:15.509060 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:15.509015    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk" podStartSLOduration=1.610102138 podStartE2EDuration="4.508995739s" totalImagesPullingTime="2.898893601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:15.508070381 +0000 UTC m=+537.882394206" watchObservedRunningTime="2026-09-09 21:42:15.508995739 +0000 UTC m=+537.883319564"
Sep 09 21:42:21.489282 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:21.489250    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 21:42:22.485184 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:22.485149    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:22.584728 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:22.584696    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:26.486289 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:26.486258    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6wpd7"
Sep 09 21:42:31.500403 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.500367    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"]
Sep 09 21:42:31.505242 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.505220    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.507580 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.507560    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 21:42:31.508285 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.508267    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 21:42:31.508400 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.508303    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 21:42:31.508400 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.508268    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wqwhc\""
Sep 09 21:42:31.511929 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.511901    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"]
Sep 09 21:42:31.523553 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.523520    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:31.648661 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.648632    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.648844 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.648671    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.648844 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.648691    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.648844 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.648710    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.648844 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.648760    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fc3502-1265-4646-be1a-69a52282ab45-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.649017 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.648873    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnq95\" (UniqueName: \"kubernetes.io/projected/10fc3502-1265-4646-be1a-69a52282ab45-kube-api-access-rnq95\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750019 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.749993    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750236 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.750028    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750236 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.750047    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750236 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.750064    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750236 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.750083    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fc3502-1265-4646-be1a-69a52282ab45-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750236 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.750144    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnq95\" (UniqueName: \"kubernetes.io/projected/10fc3502-1265-4646-be1a-69a52282ab45-kube-api-access-rnq95\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750642 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.750566    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750642 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.750617    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.750827 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.750646    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.752780 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.752756    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10fc3502-1265-4646-be1a-69a52282ab45-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.752958 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.752943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fc3502-1265-4646-be1a-69a52282ab45-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.757751 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.757728    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnq95\" (UniqueName: \"kubernetes.io/projected/10fc3502-1265-4646-be1a-69a52282ab45-kube-api-access-rnq95\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh\" (UID: \"10fc3502-1265-4646-be1a-69a52282ab45\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:31.817727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:31.817703    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:32.089898 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:32.089874    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"]
Sep 09 21:42:32.091980 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:42:32.091947    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fc3502_1265_4646_be1a_69a52282ab45.slice/crio-5e28d499915f527d425b5b573f6a11d8df6bbd16e112b489ab4ce0573681d328 WatchSource:0}: Error finding container 5e28d499915f527d425b5b573f6a11d8df6bbd16e112b489ab4ce0573681d328: Status 404 returned error can't find the container with id 5e28d499915f527d425b5b573f6a11d8df6bbd16e112b489ab4ce0573681d328
Sep 09 21:42:32.562952 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:32.562910    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh" event={"ID":"10fc3502-1265-4646-be1a-69a52282ab45","Type":"ContainerStarted","Data":"5e28d499915f527d425b5b573f6a11d8df6bbd16e112b489ab4ce0573681d328"}
Sep 09 21:42:33.569781 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:33.569696    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh" event={"ID":"10fc3502-1265-4646-be1a-69a52282ab45","Type":"ContainerStarted","Data":"ac7451e156e202505d303ae79ab1152fde8dde27bf117d85d3b6908d9124c3c4"}
Sep 09 21:42:33.586514 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:33.586471    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh" podStartSLOduration=1.500466936 podStartE2EDuration="2.58645454s" totalImagesPullingTime="1.085987604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:33.584329242 +0000 UTC m=+555.958653077" watchObservedRunningTime="2026-09-09 21:42:33.58645454 +0000 UTC m=+555.960778363"
Sep 09 21:42:41.818084 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:41.818047    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:41.818682 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:41.818123    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:41.830551 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:41.830529    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:42.620612 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:42.620583    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh"
Sep 09 21:42:52.933646 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:52.933607    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:56.322551 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:42:56.322512    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:18.018229 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:18.018176    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:18.018805 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:18.018656    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:43:18.025910 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:18.025889    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:43:28.811539 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:28.811507    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:28.911544 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:28.911511    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:30.220663 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:30.220632    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:45.474806 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.474733    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"]
Sep 09 21:43:45.478677 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.478650    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.480931 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.480904    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 21:43:45.489692 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.489664    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"]
Sep 09 21:43:45.491484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.491106    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.491484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.491156    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx59z\" (UniqueName: \"kubernetes.io/projected/528eab46-49f0-42cf-b7b7-cd837198e7d5-kube-api-access-tx59z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.491484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.491224    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528eab46-49f0-42cf-b7b7-cd837198e7d5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.491484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.491289    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.491484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.491321    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.491484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.491354    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.511940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.511917    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:45.592601 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.592570    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx59z\" (UniqueName: \"kubernetes.io/projected/528eab46-49f0-42cf-b7b7-cd837198e7d5-kube-api-access-tx59z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.592727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.592626    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528eab46-49f0-42cf-b7b7-cd837198e7d5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.592727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.592683    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.592727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.592711    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.592877 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.592741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.592877 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.592787    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.593074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.593039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.593231 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.593100    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.593231 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.593138    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.595088 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.595066    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528eab46-49f0-42cf-b7b7-cd837198e7d5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.595255 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.595237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528eab46-49f0-42cf-b7b7-cd837198e7d5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.601721 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.601696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx59z\" (UniqueName: \"kubernetes.io/projected/528eab46-49f0-42cf-b7b7-cd837198e7d5-kube-api-access-tx59z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lklhb\" (UID: \"528eab46-49f0-42cf-b7b7-cd837198e7d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:45.790930 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:45.790852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:46.063863 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:46.063837    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"]
Sep 09 21:43:46.065728 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:43:46.065699    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528eab46_49f0_42cf_b7b7_cd837198e7d5.slice/crio-a83dd13d5ee58522e7f0670cdea5c4e825015a7ae64dc01b93320b4944b605ba WatchSource:0}: Error finding container a83dd13d5ee58522e7f0670cdea5c4e825015a7ae64dc01b93320b4944b605ba: Status 404 returned error can't find the container with id a83dd13d5ee58522e7f0670cdea5c4e825015a7ae64dc01b93320b4944b605ba
Sep 09 21:43:46.067115 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:46.067097    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:43:46.895523 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:46.895489    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb" event={"ID":"528eab46-49f0-42cf-b7b7-cd837198e7d5","Type":"ContainerStarted","Data":"780f4b9bcd1ef324a0dfbea8a738a192a8add1effb177439a4941ace73cf057d"}
Sep 09 21:43:46.895906 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:46.895542    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb" event={"ID":"528eab46-49f0-42cf-b7b7-cd837198e7d5","Type":"ContainerStarted","Data":"a83dd13d5ee58522e7f0670cdea5c4e825015a7ae64dc01b93320b4944b605ba"}
Sep 09 21:43:46.911950 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:46.911901    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb" podStartSLOduration=1.73881727 podStartE2EDuration="1.911885806s" totalImagesPullingTime="173.068536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:46.910607131 +0000 UTC m=+629.284930968" watchObservedRunningTime="2026-09-09 21:43:46.911885806 +0000 UTC m=+629.286209630"
Sep 09 21:43:55.791698 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:55.791663    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:55.792292 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:55.791838    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:55.803940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:55.803921    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:43:55.952145 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:43:55.952119    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lklhb"
Sep 09 21:44:00.222354 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:44:00.222302    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:00.289624 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:44:00.289592    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:05.915912 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:44:05.915881    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:06.888017 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:44:06.887985    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:06.989375 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:44:06.989338    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:20.493697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:20.493619    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:20.594063 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:20.594026    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:30.058765 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:30.058730    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:30.157685 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:30.157646    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:31.147492 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:31.147455    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:31.247295 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:31.247264    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:46.268529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:46.268491    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:46.330017 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:46.329985    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:46.395754 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:46.395725    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:46.700669 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:46.700640    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:46.838609 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:46.838571    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:46.915267 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:46.915235    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:46.985105 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:46.985022    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:47.083762 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:47.083733    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:47.184596 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:47.184568    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:47.645579 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:47.645540    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:47.742247 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:47.742183    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:48.098097 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:48.098067    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:49.730806 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:49.730769    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:49.841059 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:49.841026    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:49.983051 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:49.982970    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:50.981084 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:50.981048    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"]
Sep 09 21:45:50.984806 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:50.984782    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:50.987086 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:50.987065    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3fa986-serving-cert\""
Sep 09 21:45:50.992021 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:50.991945    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"]
Sep 09 21:45:51.073729 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.073701    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:51.096974 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.096941    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-metrics-certs\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.097142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.096982    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.097142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.097014    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcc2k\" (UniqueName: \"kubernetes.io/projected/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-kube-api-access-rcc2k\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.097142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.097083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.157564 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.157520    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:51.198152 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.198116    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.198368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.198195    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-metrics-certs\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.198368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.198257    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.198368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.198283    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcc2k\" (UniqueName: \"kubernetes.io/projected/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-kube-api-access-rcc2k\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.198487 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:45:51.198370    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-serving-cert: secret "maas-api-e2e-worker-w4-3fa986-serving-cert" not found
Sep 09 21:45:51.198487 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:45:51.198455    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls podName:c8ed369c-b75a-4cde-ba5e-9cc8c857a675 nodeName:}" failed. No retries permitted until 2026-09-09 21:45:51.69843579 +0000 UTC m=+754.072759596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls") pod "maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" (UID: "c8ed369c-b75a-4cde-ba5e-9cc8c857a675") : secret "maas-api-e2e-worker-w4-3fa986-serving-cert" not found
Sep 09 21:45:51.198864 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.198838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.201040 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.201019    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-metrics-certs\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.210044 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.210022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcc2k\" (UniqueName: \"kubernetes.io/projected/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-kube-api-access-rcc2k\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.647775 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.647741    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:51.702966 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.702930    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:51.703125 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:45:51.703043    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-serving-cert: secret "maas-api-e2e-worker-w4-3fa986-serving-cert" not found
Sep 09 21:45:51.703125 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:45:51.703104    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls podName:c8ed369c-b75a-4cde-ba5e-9cc8c857a675 nodeName:}" failed. No retries permitted until 2026-09-09 21:45:52.703090591 +0000 UTC m=+755.077414393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls") pod "maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" (UID: "c8ed369c-b75a-4cde-ba5e-9cc8c857a675") : secret "maas-api-e2e-worker-w4-3fa986-serving-cert" not found
Sep 09 21:45:51.740379 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.740347    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"]
Sep 09 21:45:51.743933 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.743903    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:51.744378 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.744353    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:51.746031 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.746007    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3fa986\""
Sep 09 21:45:51.746166 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.746144    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3fa986-dockercfg-g9m9p\""
Sep 09 21:45:51.752267 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.752248    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"]
Sep 09 21:45:51.764976 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.764943    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"]
Sep 09 21:45:51.768068 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.768037    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:51.779803 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.779781    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"]
Sep 09 21:45:51.803699 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.803666    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwrk\" (UniqueName: \"kubernetes.io/projected/3674b913-41e2-484b-86a8-74bd53f0f0b8-kube-api-access-wmwrk\") pod \"payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c\" (UID: \"3674b913-41e2-484b-86a8-74bd53f0f0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:51.803805 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.803713    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674b913-41e2-484b-86a8-74bd53f0f0b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c\" (UID: \"3674b913-41e2-484b-86a8-74bd53f0f0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:51.803805 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.803778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtz2\" (UniqueName: \"kubernetes.io/projected/397ba615-e709-42f4-b6e0-f22cba40f113-kube-api-access-kbtz2\") pod \"payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2\" (UID: \"397ba615-e709-42f4-b6e0-f22cba40f113\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:51.803882 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.803821    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397ba615-e709-42f4-b6e0-f22cba40f113-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2\" (UID: \"397ba615-e709-42f4-b6e0-f22cba40f113\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:51.895116 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.895085    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:51.904541 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.904513    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwrk\" (UniqueName: \"kubernetes.io/projected/3674b913-41e2-484b-86a8-74bd53f0f0b8-kube-api-access-wmwrk\") pod \"payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c\" (UID: \"3674b913-41e2-484b-86a8-74bd53f0f0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:51.904671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.904571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674b913-41e2-484b-86a8-74bd53f0f0b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c\" (UID: \"3674b913-41e2-484b-86a8-74bd53f0f0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:51.904671 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.904627    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtz2\" (UniqueName: \"kubernetes.io/projected/397ba615-e709-42f4-b6e0-f22cba40f113-kube-api-access-kbtz2\") pod \"payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2\" (UID: \"397ba615-e709-42f4-b6e0-f22cba40f113\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:51.904787 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.904765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397ba615-e709-42f4-b6e0-f22cba40f113-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2\" (UID: \"397ba615-e709-42f4-b6e0-f22cba40f113\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:51.905498 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.905475    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674b913-41e2-484b-86a8-74bd53f0f0b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c\" (UID: \"3674b913-41e2-484b-86a8-74bd53f0f0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:51.905596 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.905582    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397ba615-e709-42f4-b6e0-f22cba40f113-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2\" (UID: \"397ba615-e709-42f4-b6e0-f22cba40f113\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:51.913751 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.913723    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtz2\" (UniqueName: \"kubernetes.io/projected/397ba615-e709-42f4-b6e0-f22cba40f113-kube-api-access-kbtz2\") pod \"payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2\" (UID: \"397ba615-e709-42f4-b6e0-f22cba40f113\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:51.913974 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.913954    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwrk\" (UniqueName: \"kubernetes.io/projected/3674b913-41e2-484b-86a8-74bd53f0f0b8-kube-api-access-wmwrk\") pod \"payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c\" (UID: \"3674b913-41e2-484b-86a8-74bd53f0f0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:51.999297 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:51.999238    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"]
Sep 09 21:45:52.003121 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.003100    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.005180 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.005157    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-390abc-serving-cert\""
Sep 09 21:45:52.005473 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.005450    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpr7r\" (UniqueName: \"kubernetes.io/projected/c13a3ec0-04ae-4a63-9e31-efeb966f47af-kube-api-access-gpr7r\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.005548 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.005508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-metrics-certs\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.005655 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.005641    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.005702 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.005675    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13a3ec0-04ae-4a63-9e31-efeb966f47af-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.013035 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.013015    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"]
Sep 09 21:45:52.036187 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.036160    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:52.055834 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.055808    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:52.079736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.079712    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:52.095039 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.094976    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:52.107415 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.106622    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.107415 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.106664    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13a3ec0-04ae-4a63-9e31-efeb966f47af-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.107415 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.106708    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpr7r\" (UniqueName: \"kubernetes.io/projected/c13a3ec0-04ae-4a63-9e31-efeb966f47af-kube-api-access-gpr7r\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.107415 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.106751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-metrics-certs\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.107415 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:45:52.107384    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-serving-cert: secret "maas-api-e2e-worker-w0-390abc-serving-cert" not found
Sep 09 21:45:52.107780 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:45:52.107453    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls podName:c13a3ec0-04ae-4a63-9e31-efeb966f47af nodeName:}" failed. No retries permitted until 2026-09-09 21:45:52.607432451 +0000 UTC m=+754.981756261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls") pod "maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw" (UID: "c13a3ec0-04ae-4a63-9e31-efeb966f47af") : secret "maas-api-e2e-worker-w0-390abc-serving-cert" not found
Sep 09 21:45:52.107959 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.107934    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13a3ec0-04ae-4a63-9e31-efeb966f47af-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.119529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.113565    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-metrics-certs\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.119529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.119489    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpr7r\" (UniqueName: \"kubernetes.io/projected/c13a3ec0-04ae-4a63-9e31-efeb966f47af-kube-api-access-gpr7r\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.332751 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.332727    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"]
Sep 09 21:45:52.335352 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:45:52.335322    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3674b913_41e2_484b_86a8_74bd53f0f0b8.slice/crio-830b10bf1fb327517cd576a78b4af0b518336326fb79346087b48ada409a435f WatchSource:0}: Error finding container 830b10bf1fb327517cd576a78b4af0b518336326fb79346087b48ada409a435f: Status 404 returned error can't find the container with id 830b10bf1fb327517cd576a78b4af0b518336326fb79346087b48ada409a435f
Sep 09 21:45:52.355593 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.354877    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:52.378119 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.378090    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"]
Sep 09 21:45:52.379626 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:45:52.379602    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397ba615_e709_42f4_b6e0_f22cba40f113.slice/crio-ba0de8c6f27b25ef6de7371601109c3d9e2750c76fce63228869f98eb708700c WatchSource:0}: Error finding container ba0de8c6f27b25ef6de7371601109c3d9e2750c76fce63228869f98eb708700c: Status 404 returned error can't find the container with id ba0de8c6f27b25ef6de7371601109c3d9e2750c76fce63228869f98eb708700c
Sep 09 21:45:52.422584 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.422560    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:52.472611 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.472560    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c" event={"ID":"3674b913-41e2-484b-86a8-74bd53f0f0b8","Type":"ContainerStarted","Data":"ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df"}
Sep 09 21:45:52.472893 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.472862    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c" event={"ID":"3674b913-41e2-484b-86a8-74bd53f0f0b8","Type":"ContainerStarted","Data":"830b10bf1fb327517cd576a78b4af0b518336326fb79346087b48ada409a435f"}
Sep 09 21:45:52.473016 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.472912    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:45:52.474536 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.474511    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2" event={"ID":"397ba615-e709-42f4-b6e0-f22cba40f113","Type":"ContainerStarted","Data":"cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b"}
Sep 09 21:45:52.474669 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.474553    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2" event={"ID":"397ba615-e709-42f4-b6e0-f22cba40f113","Type":"ContainerStarted","Data":"ba0de8c6f27b25ef6de7371601109c3d9e2750c76fce63228869f98eb708700c"}
Sep 09 21:45:52.474669 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.474630    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:45:52.491708 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.491668    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c" podStartSLOduration=1.491654246 podStartE2EDuration="1.491654246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:52.490920943 +0000 UTC m=+754.865244988" watchObservedRunningTime="2026-09-09 21:45:52.491654246 +0000 UTC m=+754.865978095"
Sep 09 21:45:52.509161 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.509101    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2" podStartSLOduration=1.509081165 podStartE2EDuration="1.509081165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:52.508451868 +0000 UTC m=+754.882775694" watchObservedRunningTime="2026-09-09 21:45:52.509081165 +0000 UTC m=+754.883404993"
Sep 09 21:45:52.562554 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.562519    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:52.613059 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.613022    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.615868 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.615844    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls\") pod \"maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.617707 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.617683    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:52.628772 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.628751    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:52.633320 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.633295    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"]
Sep 09 21:45:52.637651 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.637631    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:52.640129 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.639931    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-390abc\""
Sep 09 21:45:52.649597 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.647232    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"]
Sep 09 21:45:52.715033 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.714673    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca6d3a5-3b0a-48de-9220-114a74f6483e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj\" (UID: \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:52.715033 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.714716    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbgl4\" (UniqueName: \"kubernetes.io/projected/8ca6d3a5-3b0a-48de-9220-114a74f6483e-kube-api-access-sbgl4\") pod \"payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj\" (UID: \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:52.715033 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.714869    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:52.718080 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.718054    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:52.797692 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.797663    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:52.816406 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.816381    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca6d3a5-3b0a-48de-9220-114a74f6483e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj\" (UID: \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:52.816536 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.816419    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbgl4\" (UniqueName: \"kubernetes.io/projected/8ca6d3a5-3b0a-48de-9220-114a74f6483e-kube-api-access-sbgl4\") pod \"payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj\" (UID: \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:52.817112 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.817094    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca6d3a5-3b0a-48de-9220-114a74f6483e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj\" (UID: \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:52.828359 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.828331    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbgl4\" (UniqueName: \"kubernetes.io/projected/8ca6d3a5-3b0a-48de-9220-114a74f6483e-kube-api-access-sbgl4\") pod \"payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj\" (UID: \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:52.904474 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.904437    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"]
Sep 09 21:45:52.906201 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:45:52.906172    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13a3ec0_04ae_4a63_9e31_efeb966f47af.slice/crio-0e868b896ed02144d059318885effa81a4427d19e4dc42b2b3ff1f4631e1f643 WatchSource:0}: Error finding container 0e868b896ed02144d059318885effa81a4427d19e4dc42b2b3ff1f4631e1f643: Status 404 returned error can't find the container with id 0e868b896ed02144d059318885effa81a4427d19e4dc42b2b3ff1f4631e1f643
Sep 09 21:45:52.932194 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.932119    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:52.955248 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:52.955220    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:53.032791 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.032635    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:53.080746 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.080714    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"]
Sep 09 21:45:53.084362 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:45:53.084333    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ed369c_b75a_4cde_ba5e_9cc8c857a675.slice/crio-b39e913a07add516d0a3380d64834627366b10d767bfbdccf67b0d23cffc5ecd WatchSource:0}: Error finding container b39e913a07add516d0a3380d64834627366b10d767bfbdccf67b0d23cffc5ecd: Status 404 returned error can't find the container with id b39e913a07add516d0a3380d64834627366b10d767bfbdccf67b0d23cffc5ecd
Sep 09 21:45:53.132769 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.132740    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:53.236933 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.236792    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:53.251015 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.250977    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"]
Sep 09 21:45:53.253299 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:45:53.253273    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca6d3a5_3b0a_48de_9220_114a74f6483e.slice/crio-ee983782dd6d40aebff19a07b1186f055988b86bc6d7fefaf232071833da4c34 WatchSource:0}: Error finding container ee983782dd6d40aebff19a07b1186f055988b86bc6d7fefaf232071833da4c34: Status 404 returned error can't find the container with id ee983782dd6d40aebff19a07b1186f055988b86bc6d7fefaf232071833da4c34
Sep 09 21:45:53.480190 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.480153    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw" event={"ID":"c13a3ec0-04ae-4a63-9e31-efeb966f47af","Type":"ContainerStarted","Data":"0e868b896ed02144d059318885effa81a4427d19e4dc42b2b3ff1f4631e1f643"}
Sep 09 21:45:53.481664 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.481637    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" event={"ID":"c8ed369c-b75a-4cde-ba5e-9cc8c857a675","Type":"ContainerStarted","Data":"b39e913a07add516d0a3380d64834627366b10d767bfbdccf67b0d23cffc5ecd"}
Sep 09 21:45:53.483422 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.483397    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj" event={"ID":"8ca6d3a5-3b0a-48de-9220-114a74f6483e","Type":"ContainerStarted","Data":"24ba7fa3042ee17b85325c48ce01784f353e1dbd87876115e3e1d8e0d9b44daf"}
Sep 09 21:45:53.483519 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.483447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj" event={"ID":"8ca6d3a5-3b0a-48de-9220-114a74f6483e","Type":"ContainerStarted","Data":"ee983782dd6d40aebff19a07b1186f055988b86bc6d7fefaf232071833da4c34"}
Sep 09 21:45:53.483519 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.483505    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:45:53.498016 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.497759    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj" podStartSLOduration=1.497740608 podStartE2EDuration="1.497740608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:53.497240966 +0000 UTC m=+755.871564792" watchObservedRunningTime="2026-09-09 21:45:53.497740608 +0000 UTC m=+755.872064430"
Sep 09 21:45:53.836681 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.836607    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:53.931526 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:53.931503    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:54.033604 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:54.033565    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:54.489974 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:54.489930    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw" event={"ID":"c13a3ec0-04ae-4a63-9e31-efeb966f47af","Type":"ContainerStarted","Data":"0ceb6614396140cfb9a44dfb58b6a3eadaf4bf4a0107bba89cc4ca5f6359c639"}
Sep 09 21:45:54.490153 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:54.490000    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:45:54.491773 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:54.491748    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" event={"ID":"c8ed369c-b75a-4cde-ba5e-9cc8c857a675","Type":"ContainerStarted","Data":"d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355"}
Sep 09 21:45:54.491877 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:54.491815    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:45:54.505220 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:54.505157    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw" podStartSLOduration=2.928227341 podStartE2EDuration="3.505141662s" totalImagesPullingTime="576.914321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:54.50409918 +0000 UTC m=+756.878423004" watchObservedRunningTime="2026-09-09 21:45:54.505141662 +0000 UTC m=+756.879465486"
Sep 09 21:45:54.520640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:54.520595    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" podStartSLOduration=3.677619604 podStartE2EDuration="4.520583653s" totalImagesPullingTime="842.964049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:54.518559641 +0000 UTC m=+756.892883472" watchObservedRunningTime="2026-09-09 21:45:54.520583653 +0000 UTC m=+756.894907478"
Sep 09 21:45:54.793630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:54.793548    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:55.116456 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:55.116377    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:55.213998 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:55.213963    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:55.322992 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:55.322951    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:55.428565 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:55.428536    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:55.636576 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:55.636542    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:55.834010 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:55.833922    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:56.817690 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:56.817652    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:56.886131 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:56.886091    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:57.911792 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.911760    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"]
Sep 09 21:45:57.915509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.915489    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:57.917659 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.917640    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-66cedd-serving-cert\""
Sep 09 21:45:57.925478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.925455    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"]
Sep 09 21:45:57.967653 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.967619    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-maas-api-tls\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:57.967813 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.967710    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5ql\" (UniqueName: \"kubernetes.io/projected/3f2be80d-36cc-4489-8eb7-a77246c44f5c-kube-api-access-wn5ql\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:57.967813 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.967748    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2be80d-36cc-4489-8eb7-a77246c44f5c-openshift-service-ca\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:57.967910 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.967821    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-metrics-certs\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:57.974336 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:57.974309    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:58.068773 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.068744    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-maas-api-tls\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.068889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.068797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5ql\" (UniqueName: \"kubernetes.io/projected/3f2be80d-36cc-4489-8eb7-a77246c44f5c-kube-api-access-wn5ql\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.068946 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.068922    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2be80d-36cc-4489-8eb7-a77246c44f5c-openshift-service-ca\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.068995 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.068981    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-metrics-certs\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.069566 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.069543    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2be80d-36cc-4489-8eb7-a77246c44f5c-openshift-service-ca\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.071395 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.071368    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-metrics-certs\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.071395 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.071378    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-maas-api-tls\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.078396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.078369    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:58.080860 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.080837    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5ql\" (UniqueName: \"kubernetes.io/projected/3f2be80d-36cc-4489-8eb7-a77246c44f5c-kube-api-access-wn5ql\") pod \"maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.204600 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.204450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:58.227530 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.227506    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:58.498965 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.498942    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"]
Sep 09 21:45:58.501112 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:45:58.501080    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2be80d_36cc_4489_8eb7_a77246c44f5c.slice/crio-35932fa3dc24afe89e8a99d5af4c54efed786876387ce019af5fb2a248cd5893 WatchSource:0}: Error finding container 35932fa3dc24afe89e8a99d5af4c54efed786876387ce019af5fb2a248cd5893: Status 404 returned error can't find the container with id 35932fa3dc24afe89e8a99d5af4c54efed786876387ce019af5fb2a248cd5893
Sep 09 21:45:58.513517 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.513488    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz" event={"ID":"3f2be80d-36cc-4489-8eb7-a77246c44f5c","Type":"ContainerStarted","Data":"35932fa3dc24afe89e8a99d5af4c54efed786876387ce019af5fb2a248cd5893"}
Sep 09 21:45:58.603283 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.603247    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:58.676736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.676702    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:58.678034 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.677987    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"]
Sep 09 21:45:58.682984 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.682965    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:58.685437 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.685409    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-66cedd\""
Sep 09 21:45:58.685786 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.685762    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-66cedd-dockercfg-h5rtc\""
Sep 09 21:45:58.688746 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.688722    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"]
Sep 09 21:45:58.742682 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.742653    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"]
Sep 09 21:45:58.746170 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.746150    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:58.753694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.753670    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"]
Sep 09 21:45:58.775319 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.775298    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990459a4-88aa-4333-81c1-83f16df14b27-plugins-config-volume\") pod \"payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs\" (UID: \"990459a4-88aa-4333-81c1-83f16df14b27\") " pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:58.775421 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.775341    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899bd60a-dd42-4663-a771-0d44e4ebc153-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn\" (UID: \"899bd60a-dd42-4663-a771-0d44e4ebc153\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:58.775421 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.775368    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqd9c\" (UniqueName: \"kubernetes.io/projected/990459a4-88aa-4333-81c1-83f16df14b27-kube-api-access-dqd9c\") pod \"payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs\" (UID: \"990459a4-88aa-4333-81c1-83f16df14b27\") " pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:58.775421 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.775387    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmshl\" (UniqueName: \"kubernetes.io/projected/899bd60a-dd42-4663-a771-0d44e4ebc153-kube-api-access-gmshl\") pod \"payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn\" (UID: \"899bd60a-dd42-4663-a771-0d44e4ebc153\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:58.876394 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.876358    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990459a4-88aa-4333-81c1-83f16df14b27-plugins-config-volume\") pod \"payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs\" (UID: \"990459a4-88aa-4333-81c1-83f16df14b27\") " pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:58.876633 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.876405    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899bd60a-dd42-4663-a771-0d44e4ebc153-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn\" (UID: \"899bd60a-dd42-4663-a771-0d44e4ebc153\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:58.876633 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.876432    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqd9c\" (UniqueName: \"kubernetes.io/projected/990459a4-88aa-4333-81c1-83f16df14b27-kube-api-access-dqd9c\") pod \"payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs\" (UID: \"990459a4-88aa-4333-81c1-83f16df14b27\") " pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:58.876633 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.876460    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmshl\" (UniqueName: \"kubernetes.io/projected/899bd60a-dd42-4663-a771-0d44e4ebc153-kube-api-access-gmshl\") pod \"payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn\" (UID: \"899bd60a-dd42-4663-a771-0d44e4ebc153\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:58.877178 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.877153    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899bd60a-dd42-4663-a771-0d44e4ebc153-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn\" (UID: \"899bd60a-dd42-4663-a771-0d44e4ebc153\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:58.877298 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.877153    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990459a4-88aa-4333-81c1-83f16df14b27-plugins-config-volume\") pod \"payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs\" (UID: \"990459a4-88aa-4333-81c1-83f16df14b27\") " pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:58.884622 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.884601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqd9c\" (UniqueName: \"kubernetes.io/projected/990459a4-88aa-4333-81c1-83f16df14b27-kube-api-access-dqd9c\") pod \"payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs\" (UID: \"990459a4-88aa-4333-81c1-83f16df14b27\") " pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:58.884720 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.884704    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmshl\" (UniqueName: \"kubernetes.io/projected/899bd60a-dd42-4663-a771-0d44e4ebc153-kube-api-access-gmshl\") pod \"payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn\" (UID: \"899bd60a-dd42-4663-a771-0d44e4ebc153\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:58.995856 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:58.995827    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:59.056961 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.056575    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:59.270833 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.270808    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"]
Sep 09 21:45:59.272858 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:45:59.272830    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899bd60a_dd42_4663_a771_0d44e4ebc153.slice/crio-c8668518f2ee9178a0cde1642dbf7f29226cd02d458f70414db4463a547252f4 WatchSource:0}: Error finding container c8668518f2ee9178a0cde1642dbf7f29226cd02d458f70414db4463a547252f4: Status 404 returned error can't find the container with id c8668518f2ee9178a0cde1642dbf7f29226cd02d458f70414db4463a547252f4
Sep 09 21:45:59.336264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.336239    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"]
Sep 09 21:45:59.339603 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:45:59.339573    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990459a4_88aa_4333_81c1_83f16df14b27.slice/crio-4a2e7abe804584d0c1c75032d05c72458c0b0d7f629186111e776b73733901c1 WatchSource:0}: Error finding container 4a2e7abe804584d0c1c75032d05c72458c0b0d7f629186111e776b73733901c1: Status 404 returned error can't find the container with id 4a2e7abe804584d0c1c75032d05c72458c0b0d7f629186111e776b73733901c1
Sep 09 21:45:59.520497 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.520400    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz" event={"ID":"3f2be80d-36cc-4489-8eb7-a77246c44f5c","Type":"ContainerStarted","Data":"a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345"}
Sep 09 21:45:59.520497 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.520486    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:45:59.522196 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.522163    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn" event={"ID":"899bd60a-dd42-4663-a771-0d44e4ebc153","Type":"ContainerStarted","Data":"2f9670a07e397493dd42ad9c011f0ed31167d06335d6ebdd6998b85f568ae57a"}
Sep 09 21:45:59.522368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.522237    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:45:59.522368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.522254    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn" event={"ID":"899bd60a-dd42-4663-a771-0d44e4ebc153","Type":"ContainerStarted","Data":"c8668518f2ee9178a0cde1642dbf7f29226cd02d458f70414db4463a547252f4"}
Sep 09 21:45:59.523958 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.523931    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs" event={"ID":"990459a4-88aa-4333-81c1-83f16df14b27","Type":"ContainerStarted","Data":"ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483"}
Sep 09 21:45:59.524078 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.523973    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs" event={"ID":"990459a4-88aa-4333-81c1-83f16df14b27","Type":"ContainerStarted","Data":"4a2e7abe804584d0c1c75032d05c72458c0b0d7f629186111e776b73733901c1"}
Sep 09 21:45:59.524078 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.523997    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:45:59.568770 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.568721    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz" podStartSLOduration=2.163686376 podStartE2EDuration="2.568701007s" totalImagesPullingTime="405.014631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:59.564721417 +0000 UTC m=+761.939045243" watchObservedRunningTime="2026-09-09 21:45:59.568701007 +0000 UTC m=+761.943024834"
Sep 09 21:45:59.634824 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.634766    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs" podStartSLOduration=1.634750264 podStartE2EDuration="1.634750264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:59.63417692 +0000 UTC m=+762.008500745" watchObservedRunningTime="2026-09-09 21:45:59.634750264 +0000 UTC m=+762.009074087"
Sep 09 21:45:59.646379 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.646345    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:59.668376 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.668309    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn" podStartSLOduration=1.668287973 podStartE2EDuration="1.668287973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:59.665157228 +0000 UTC m=+762.039481059" watchObservedRunningTime="2026-09-09 21:45:59.668287973 +0000 UTC m=+762.042611800"
Sep 09 21:45:59.740550 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:45:59.740509    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:00.503408 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.503380    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:46:00.503847 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.503830    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 21:46:00.788970 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.788894    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"]
Sep 09 21:46:00.792900 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.792881    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.794958 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.794934    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-04d63732-serving-cert\""
Sep 09 21:46:00.804250 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.804226    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"]
Sep 09 21:46:00.824687 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.824662    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:00.895481 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.895451    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94e94d8-d527-446d-847b-89f922697598-openshift-service-ca\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.895629 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.895557    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-maas-api-tls\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.895629 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.895611    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp82l\" (UniqueName: \"kubernetes.io/projected/c94e94d8-d527-446d-847b-89f922697598-kube-api-access-fp82l\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.895741 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.895631    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-metrics-certs\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.925471 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.925411    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:00.996417 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.996383    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp82l\" (UniqueName: \"kubernetes.io/projected/c94e94d8-d527-446d-847b-89f922697598-kube-api-access-fp82l\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.996595 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.996421    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-metrics-certs\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.996595 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.996461    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94e94d8-d527-446d-847b-89f922697598-openshift-service-ca\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.996595 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.996557    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-maas-api-tls\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.997173 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.997152    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94e94d8-d527-446d-847b-89f922697598-openshift-service-ca\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.999069 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.999042    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-maas-api-tls\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:00.999169 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:00.999048    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-metrics-certs\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:01.003467 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:01.003449    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp82l\" (UniqueName: \"kubernetes.io/projected/c94e94d8-d527-446d-847b-89f922697598-kube-api-access-fp82l\") pod \"maas-api-e2e-ait-04d63732-786cd5cf54-rz67b\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:01.030990 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:01.030963    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:01.107544 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:01.107471    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:01.344783 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:01.344715    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:01.383093 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:01.383028    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"]
Sep 09 21:46:01.385460 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:46:01.385431    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94e94d8_d527_446d_847b_89f922697598.slice/crio-cda0cd26b927e81ffc3adfa01b36226be87a45e38e0b328fc48399dd59675dad WatchSource:0}: Error finding container cda0cd26b927e81ffc3adfa01b36226be87a45e38e0b328fc48399dd59675dad: Status 404 returned error can't find the container with id cda0cd26b927e81ffc3adfa01b36226be87a45e38e0b328fc48399dd59675dad
Sep 09 21:46:01.424130 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:01.424098    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:01.538178 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:01.538147    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b" event={"ID":"c94e94d8-d527-446d-847b-89f922697598","Type":"ContainerStarted","Data":"cda0cd26b927e81ffc3adfa01b36226be87a45e38e0b328fc48399dd59675dad"}
Sep 09 21:46:02.545942 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:02.545861    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b" event={"ID":"c94e94d8-d527-446d-847b-89f922697598","Type":"ContainerStarted","Data":"06aea9e51814ab76bec5b24b55af7d46cca319b21e5c8f987d9a323bfb3cdb3e"}
Sep 09 21:46:02.545942 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:02.545937    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:02.562081 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:02.562031    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b" podStartSLOduration=1.656992232 podStartE2EDuration="2.562015699s" totalImagesPullingTime="905.023467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:02.560746124 +0000 UTC m=+764.935069949" watchObservedRunningTime="2026-09-09 21:46:02.562015699 +0000 UTC m=+764.936339547"
Sep 09 21:46:03.485275 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:03.485244    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 21:46:03.485502 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:03.485486    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 21:46:04.493691 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:04.493662    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:46:05.165072 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:05.165042    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:05.316530 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:05.316491    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:05.417540 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:05.417455    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:05.516681 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:05.516644    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:05.536774 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:05.536746    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:46:08.179219 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:08.179186    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:08.558092 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:08.558013    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:09.704563 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:09.704531    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:10.533293 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:10.533265    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:46:10.533472 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:10.533344    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:46:11.577902 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:11.577872    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:11.693151 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:11.693117    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:11.826858 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:11.826815    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:12.197343 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:12.197313    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:13.295612 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:13.295577    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:13.991989 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:13.991954    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:14.610942 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:14.610888    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:14.714512 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:14.714480    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:15.091120 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:15.091093    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:15.236127 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:15.236081    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:15.610379 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:15.610344    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:17.686896 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:17.686847    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:17.788700 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:17.788664    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:17.886652 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:17.886623    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:18.056138 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:18.056059    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:18.957272 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:18.957242    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:19.779871 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:19.779841    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:20.090723 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:20.090635    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:20.391384 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:20.391299    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:20.762963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:20.762909    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:21.635120 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:21.635090    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:21.735848 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:21.735810    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:23.100411 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:23.100375    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:23.196881 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:23.196849    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:24.034874 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.034842    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:24.143978 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.143925    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:24.275534 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.275498    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"]
Sep 09 21:46:24.279877 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.279856    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.283135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.283114    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-da0a18-serving-cert\""
Sep 09 21:46:24.304515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.304490    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"]
Sep 09 21:46:24.314710 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.314684    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:24.406630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.406576    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:24.427073 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.427045    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-maas-api-tls\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.427073 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.427077    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzqm\" (UniqueName: \"kubernetes.io/projected/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-kube-api-access-xzzqm\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.427349 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.427145    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-metrics-certs\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.427349 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.427194    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.527864 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.527831    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-metrics-certs\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.528073 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.527906    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.528073 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.527976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-maas-api-tls\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.528073 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.528006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzqm\" (UniqueName: \"kubernetes.io/projected/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-kube-api-access-xzzqm\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.528770 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.528745    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.530749 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.530724    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-metrics-certs\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.531431 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.531409    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-maas-api-tls\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.539526 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.539468    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzqm\" (UniqueName: \"kubernetes.io/projected/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-kube-api-access-xzzqm\") pod \"maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.543746 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.543721    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:24.593359 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.593331    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:24.803054 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.802971    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:24.875911 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.875876    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:24.877140 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:24.877111    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"]
Sep 09 21:46:24.878873 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:46:24.878849    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de6c8e4_0d4c_44e4_9a97_456fdce81fbd.slice/crio-c40c87e492392c13e67bb206e452b71fd0ee65c1d9ffe47aff2e50f8a2d13ef4 WatchSource:0}: Error finding container c40c87e492392c13e67bb206e452b71fd0ee65c1d9ffe47aff2e50f8a2d13ef4: Status 404 returned error can't find the container with id c40c87e492392c13e67bb206e452b71fd0ee65c1d9ffe47aff2e50f8a2d13ef4
Sep 09 21:46:25.312002 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:25.311946    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:25.671808 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:25.671776    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2" event={"ID":"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd","Type":"ContainerStarted","Data":"afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906"}
Sep 09 21:46:25.671974 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:25.671834    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2" event={"ID":"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd","Type":"ContainerStarted","Data":"c40c87e492392c13e67bb206e452b71fd0ee65c1d9ffe47aff2e50f8a2d13ef4"}
Sep 09 21:46:25.671974 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:25.671857    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:25.689295 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:25.689247    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2" podStartSLOduration=1.3192034449999999 podStartE2EDuration="1.689231825s" totalImagesPullingTime="370.02838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:25.687864451 +0000 UTC m=+788.062188274" watchObservedRunningTime="2026-09-09 21:46:25.689231825 +0000 UTC m=+788.063555643"
Sep 09 21:46:25.985457 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:25.985367    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:31.682751 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:31.682723    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:46:38.655988 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:38.655957    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:38.709579 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:38.709550    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:41.005642 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.005606    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:41.551411 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.551382    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"]
Sep 09 21:46:41.551668 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.551638    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b" podUID="c94e94d8-d527-446d-847b-89f922697598" containerName="maas-api" containerID="cri-o://06aea9e51814ab76bec5b24b55af7d46cca319b21e5c8f987d9a323bfb3cdb3e" gracePeriod=30
Sep 09 21:46:41.564587 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.564534    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:41.631354 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.631324    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:41.704733 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.704702    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:41.764527 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.764494    2585 generic.go:356] "Generic (PLEG): container finished" podID="c94e94d8-d527-446d-847b-89f922697598" containerID="06aea9e51814ab76bec5b24b55af7d46cca319b21e5c8f987d9a323bfb3cdb3e" exitCode=0
Sep 09 21:46:41.764527 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.764517    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b" event={"ID":"c94e94d8-d527-446d-847b-89f922697598","Type":"ContainerDied","Data":"06aea9e51814ab76bec5b24b55af7d46cca319b21e5c8f987d9a323bfb3cdb3e"}
Sep 09 21:46:41.771987 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.771916    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:41.805600 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.805538    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:41.884223 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.884173    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94e94d8-d527-446d-847b-89f922697598-openshift-service-ca\") pod \"c94e94d8-d527-446d-847b-89f922697598\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") "
Sep 09 21:46:41.884223 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.884226    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-maas-api-tls\") pod \"c94e94d8-d527-446d-847b-89f922697598\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") "
Sep 09 21:46:41.884542 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.884269    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-metrics-certs\") pod \"c94e94d8-d527-446d-847b-89f922697598\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") "
Sep 09 21:46:41.884542 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.884343    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp82l\" (UniqueName: \"kubernetes.io/projected/c94e94d8-d527-446d-847b-89f922697598-kube-api-access-fp82l\") pod \"c94e94d8-d527-446d-847b-89f922697598\" (UID: \"c94e94d8-d527-446d-847b-89f922697598\") "
Sep 09 21:46:41.884542 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.884504    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94e94d8-d527-446d-847b-89f922697598-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c94e94d8-d527-446d-847b-89f922697598" (UID: "c94e94d8-d527-446d-847b-89f922697598"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:41.884723 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.884693    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94e94d8-d527-446d-847b-89f922697598-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:46:41.887460 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.887421    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94e94d8-d527-446d-847b-89f922697598-kube-api-access-fp82l" (OuterVolumeSpecName: "kube-api-access-fp82l") pod "c94e94d8-d527-446d-847b-89f922697598" (UID: "c94e94d8-d527-446d-847b-89f922697598"). InnerVolumeSpecName "kube-api-access-fp82l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:46:41.888779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.888745    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c94e94d8-d527-446d-847b-89f922697598" (UID: "c94e94d8-d527-446d-847b-89f922697598"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:46:41.888875 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.888844    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c94e94d8-d527-446d-847b-89f922697598" (UID: "c94e94d8-d527-446d-847b-89f922697598"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:46:41.892014 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.891987    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:41.985384 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.985354    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:46:41.985384 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.985382    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94e94d8-d527-446d-847b-89f922697598-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:46:41.985588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:41.985394    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp82l\" (UniqueName: \"kubernetes.io/projected/c94e94d8-d527-446d-847b-89f922697598-kube-api-access-fp82l\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:46:42.357080 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.357039    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:42.452339 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.452307    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:42.554920 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.554886    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:42.771239 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.771150    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"
Sep 09 21:46:42.771239 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.771178    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b" event={"ID":"c94e94d8-d527-446d-847b-89f922697598","Type":"ContainerDied","Data":"cda0cd26b927e81ffc3adfa01b36226be87a45e38e0b328fc48399dd59675dad"}
Sep 09 21:46:42.771484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.771264    2585 scope.go:117] "RemoveContainer" containerID="06aea9e51814ab76bec5b24b55af7d46cca319b21e5c8f987d9a323bfb3cdb3e"
Sep 09 21:46:42.791964 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.791942    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"]
Sep 09 21:46:42.794960 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.794934    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04d63732-786cd5cf54-rz67b"]
Sep 09 21:46:42.884896 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:42.884859    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:43.465468 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:43.465421    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:44.127911 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:44.127880    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c94e94d8-d527-446d-847b-89f922697598" path="/var/lib/kubelet/pods/c94e94d8-d527-446d-847b-89f922697598/volumes"
Sep 09 21:46:45.185790 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:45.185758    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:46.070727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.070691    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:46.170396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.170359    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:46.461364 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.461336    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"]
Sep 09 21:46:46.461838 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.461742    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c94e94d8-d527-446d-847b-89f922697598" containerName="maas-api"
Sep 09 21:46:46.461838 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.461754    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c94e94d8-d527-446d-847b-89f922697598" containerName="maas-api"
Sep 09 21:46:46.461838 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.461827    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c94e94d8-d527-446d-847b-89f922697598" containerName="maas-api"
Sep 09 21:46:46.466610 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.466587    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.468924 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.468907    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-98c7fb47-serving-cert\""
Sep 09 21:46:46.476451 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.476430    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"]
Sep 09 21:46:46.495998 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.495974    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:46.528425 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.528392    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.528567 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.528436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-metrics-certs\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.528567 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.528508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9rhl\" (UniqueName: \"kubernetes.io/projected/c453e7b5-defc-40f3-9a13-eacc944c504e-kube-api-access-z9rhl\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.528656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.528631    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453e7b5-defc-40f3-9a13-eacc944c504e-openshift-service-ca\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.551541 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.551516    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:46.629823 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.629793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453e7b5-defc-40f3-9a13-eacc944c504e-openshift-service-ca\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.630005 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.629860    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.630005 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.629892    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-metrics-certs\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.630005 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.629916    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rhl\" (UniqueName: \"kubernetes.io/projected/c453e7b5-defc-40f3-9a13-eacc944c504e-kube-api-access-z9rhl\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.630005 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:46:46.629972    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-serving-cert: secret "maas-api-e2e-migrate-98c7fb47-serving-cert" not found
Sep 09 21:46:46.630282 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:46:46.630038    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls podName:c453e7b5-defc-40f3-9a13-eacc944c504e nodeName:}" failed. No retries permitted until 2026-09-09 21:46:47.130016313 +0000 UTC m=+809.504340116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls") pod "maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf" (UID: "c453e7b5-defc-40f3-9a13-eacc944c504e") : secret "maas-api-e2e-migrate-98c7fb47-serving-cert" not found
Sep 09 21:46:46.630523 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.630507    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453e7b5-defc-40f3-9a13-eacc944c504e-openshift-service-ca\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.632553 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.632530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-metrics-certs\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.637934 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.637906    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rhl\" (UniqueName: \"kubernetes.io/projected/c453e7b5-defc-40f3-9a13-eacc944c504e-kube-api-access-z9rhl\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:46.801486 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.801411    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:46.969661 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:46.969620    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:47.066371 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:47.066297    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:47.133836 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:47.133807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:47.136467 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:47.136432    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls\") pod \"maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:47.381261 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:47.381159    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:47.393550 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:47.393517    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:47.663457 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:47.663432    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"]
Sep 09 21:46:47.666063 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:46:47.666033    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc453e7b5_defc_40f3_9a13_eacc944c504e.slice/crio-11b26b2dfc5e4830284b8f84abbc1cfb08baa6531f7da1ab42ba161bc1bc3dd2 WatchSource:0}: Error finding container 11b26b2dfc5e4830284b8f84abbc1cfb08baa6531f7da1ab42ba161bc1bc3dd2: Status 404 returned error can't find the container with id 11b26b2dfc5e4830284b8f84abbc1cfb08baa6531f7da1ab42ba161bc1bc3dd2
Sep 09 21:46:47.796630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:47.796595    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf" event={"ID":"c453e7b5-defc-40f3-9a13-eacc944c504e","Type":"ContainerStarted","Data":"11b26b2dfc5e4830284b8f84abbc1cfb08baa6531f7da1ab42ba161bc1bc3dd2"}
Sep 09 21:46:48.018459 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:48.018419    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:48.231587 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:48.231468    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:48.803049 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:48.803014    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf" event={"ID":"c453e7b5-defc-40f3-9a13-eacc944c504e","Type":"ContainerStarted","Data":"5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff"}
Sep 09 21:46:48.803455 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:48.803103    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:46:48.821891 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:48.821847    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf" podStartSLOduration=2.480412264 podStartE2EDuration="2.821833509s" totalImagesPullingTime="341.421245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:48.819774655 +0000 UTC m=+811.194098491" watchObservedRunningTime="2026-09-09 21:46:48.821833509 +0000 UTC m=+811.196157372"
Sep 09 21:46:54.814905 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:46:54.814864    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:47:05.451698 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:05.451664    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:17.470553 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.470518    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"]
Sep 09 21:47:17.470973 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.470779    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf" podUID="c453e7b5-defc-40f3-9a13-eacc944c504e" containerName="maas-api" containerID="cri-o://5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff" gracePeriod=30
Sep 09 21:47:17.526644 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.526582    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:17.623425 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.623395    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:17.729384 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.729356    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:17.750734 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.750699    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:47:17.804912 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.804881    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9rhl\" (UniqueName: \"kubernetes.io/projected/c453e7b5-defc-40f3-9a13-eacc944c504e-kube-api-access-z9rhl\") pod \"c453e7b5-defc-40f3-9a13-eacc944c504e\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") "
Sep 09 21:47:17.805089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.804983    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls\") pod \"c453e7b5-defc-40f3-9a13-eacc944c504e\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") "
Sep 09 21:47:17.805089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.805020    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453e7b5-defc-40f3-9a13-eacc944c504e-openshift-service-ca\") pod \"c453e7b5-defc-40f3-9a13-eacc944c504e\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") "
Sep 09 21:47:17.805089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.805048    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-metrics-certs\") pod \"c453e7b5-defc-40f3-9a13-eacc944c504e\" (UID: \"c453e7b5-defc-40f3-9a13-eacc944c504e\") "
Sep 09 21:47:17.805404 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.805377    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c453e7b5-defc-40f3-9a13-eacc944c504e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c453e7b5-defc-40f3-9a13-eacc944c504e" (UID: "c453e7b5-defc-40f3-9a13-eacc944c504e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:47:17.807179 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.807153    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c453e7b5-defc-40f3-9a13-eacc944c504e-kube-api-access-z9rhl" (OuterVolumeSpecName: "kube-api-access-z9rhl") pod "c453e7b5-defc-40f3-9a13-eacc944c504e" (UID: "c453e7b5-defc-40f3-9a13-eacc944c504e"). InnerVolumeSpecName "kube-api-access-z9rhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:17.807362 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.807337    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c453e7b5-defc-40f3-9a13-eacc944c504e" (UID: "c453e7b5-defc-40f3-9a13-eacc944c504e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:47:17.807416 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.807365    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c453e7b5-defc-40f3-9a13-eacc944c504e" (UID: "c453e7b5-defc-40f3-9a13-eacc944c504e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:47:17.833120 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.833093    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:17.906057 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.906027    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9rhl\" (UniqueName: \"kubernetes.io/projected/c453e7b5-defc-40f3-9a13-eacc944c504e-kube-api-access-z9rhl\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:47:17.906057 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.906059    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:47:17.906057 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.906074    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453e7b5-defc-40f3-9a13-eacc944c504e-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:47:17.906383 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.906088    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c453e7b5-defc-40f3-9a13-eacc944c504e-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:47:17.938012 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.937983    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:17.951622 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.951591    2585 generic.go:356] "Generic (PLEG): container finished" podID="c453e7b5-defc-40f3-9a13-eacc944c504e" containerID="5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff" exitCode=0
Sep 09 21:47:17.951758 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.951641    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"
Sep 09 21:47:17.951758 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.951643    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf" event={"ID":"c453e7b5-defc-40f3-9a13-eacc944c504e","Type":"ContainerDied","Data":"5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff"}
Sep 09 21:47:17.951758 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.951695    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf" event={"ID":"c453e7b5-defc-40f3-9a13-eacc944c504e","Type":"ContainerDied","Data":"11b26b2dfc5e4830284b8f84abbc1cfb08baa6531f7da1ab42ba161bc1bc3dd2"}
Sep 09 21:47:17.951758 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.951717    2585 scope.go:117] "RemoveContainer" containerID="5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff"
Sep 09 21:47:17.960936 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.960916    2585 scope.go:117] "RemoveContainer" containerID="5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff"
Sep 09 21:47:17.961219 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:47:17.961184    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff\": container with ID starting with 5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff not found: ID does not exist" containerID="5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff"
Sep 09 21:47:17.961293 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.961226    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff"} err="failed to get container status \"5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff\": rpc error: code = NotFound desc = could not find container \"5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff\": container with ID starting with 5a99b726bc20fd41df5ed4f1049665dc24622b706332ea6d8acac55fee0c97ff not found: ID does not exist"
Sep 09 21:47:17.974123 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.974099    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"]
Sep 09 21:47:17.978637 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:17.978583    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98c7fb47-7c65b746dd-hrcdf"]
Sep 09 21:47:18.129889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:18.129860    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c453e7b5-defc-40f3-9a13-eacc944c504e" path="/var/lib/kubelet/pods/c453e7b5-defc-40f3-9a13-eacc944c504e/volumes"
Sep 09 21:47:18.361446 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:18.361351    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:18.778581 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:18.778539    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:18.876703 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:18.876673    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:20.998518 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:20.998488    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:21.097082 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:21.097047    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:22.108277 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.108244    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"]
Sep 09 21:47:22.108691 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.108642    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c453e7b5-defc-40f3-9a13-eacc944c504e" containerName="maas-api"
Sep 09 21:47:22.108691 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.108653    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c453e7b5-defc-40f3-9a13-eacc944c504e" containerName="maas-api"
Sep 09 21:47:22.108789 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.108728    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c453e7b5-defc-40f3-9a13-eacc944c504e" containerName="maas-api"
Sep 09 21:47:22.113630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.113607    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.117111 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.116870    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b27eb4af-serving-cert\""
Sep 09 21:47:22.131719 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.131635    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"]
Sep 09 21:47:22.143005 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.142973    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd69dc49-a23c-4dd9-af84-525dca03041d-openshift-service-ca\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.143126 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.143079    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-metrics-certs\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.143191 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.143177    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.143354 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.143335    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nxn\" (UniqueName: \"kubernetes.io/projected/fd69dc49-a23c-4dd9-af84-525dca03041d-kube-api-access-z4nxn\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.174606 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.174583    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:22.244432 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.244403    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nxn\" (UniqueName: \"kubernetes.io/projected/fd69dc49-a23c-4dd9-af84-525dca03041d-kube-api-access-z4nxn\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.244580 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.244456    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd69dc49-a23c-4dd9-af84-525dca03041d-openshift-service-ca\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.244580 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.244507    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-metrics-certs\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.244580 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.244532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.244698 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:47:22.244647    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-serving-cert: secret "maas-api-e2e-ait-b27eb4af-serving-cert" not found
Sep 09 21:47:22.244738 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:47:22.244714    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls podName:fd69dc49-a23c-4dd9-af84-525dca03041d nodeName:}" failed. No retries permitted until 2026-09-09 21:47:22.744692692 +0000 UTC m=+845.119016495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls") pod "maas-api-e2e-ait-b27eb4af-76cc474494-trlm5" (UID: "fd69dc49-a23c-4dd9-af84-525dca03041d") : secret "maas-api-e2e-ait-b27eb4af-serving-cert" not found
Sep 09 21:47:22.245400 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.245229    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd69dc49-a23c-4dd9-af84-525dca03041d-openshift-service-ca\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.247060 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.247039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-metrics-certs\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.253903 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.253878    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nxn\" (UniqueName: \"kubernetes.io/projected/fd69dc49-a23c-4dd9-af84-525dca03041d-kube-api-access-z4nxn\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.277075 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.277051    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:22.748575 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.748542    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.751200 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.751173    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls\") pod \"maas-api-e2e-ait-b27eb4af-76cc474494-trlm5\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:22.810162 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.810132    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:22.915633 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:22.915595    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:23.026964 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:23.026887    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:23.315131 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:23.315106    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"]
Sep 09 21:47:23.317100 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:47:23.317060    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd69dc49_a23c_4dd9_af84_525dca03041d.slice/crio-312fda78c7f6a87d0fe1cebe6eba2b874d0179d20e07e71b37a51cb8d4dcd276 WatchSource:0}: Error finding container 312fda78c7f6a87d0fe1cebe6eba2b874d0179d20e07e71b37a51cb8d4dcd276: Status 404 returned error can't find the container with id 312fda78c7f6a87d0fe1cebe6eba2b874d0179d20e07e71b37a51cb8d4dcd276
Sep 09 21:47:23.983469 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:23.983433    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5" event={"ID":"fd69dc49-a23c-4dd9-af84-525dca03041d","Type":"ContainerStarted","Data":"312fda78c7f6a87d0fe1cebe6eba2b874d0179d20e07e71b37a51cb8d4dcd276"}
Sep 09 21:47:24.990641 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:24.990603    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5" event={"ID":"fd69dc49-a23c-4dd9-af84-525dca03041d","Type":"ContainerStarted","Data":"5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca"}
Sep 09 21:47:24.991103 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:24.990738    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:25.012780 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:25.012727    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5" podStartSLOduration=2.240333012 podStartE2EDuration="3.012713189s" totalImagesPullingTime="772.380177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:25.0112753 +0000 UTC m=+847.385599126" watchObservedRunningTime="2026-09-09 21:47:25.012713189 +0000 UTC m=+847.387037014"
Sep 09 21:47:31.002087 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:31.002056    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:33.338062 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:33.338027    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:43.238488 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:43.238443    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:43.508285 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:43.508166    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:52.224091 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:52.224061    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:53.553830 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.553800    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"]
Sep 09 21:47:53.554233 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.553994    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5" podUID="fd69dc49-a23c-4dd9-af84-525dca03041d" containerName="maas-api" containerID="cri-o://5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca" gracePeriod=30
Sep 09 21:47:53.571489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.571463    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:53.739121 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.739092    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:53.814020 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.813957    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:53.842326 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.842295    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd69dc49-a23c-4dd9-af84-525dca03041d-openshift-service-ca\") pod \"fd69dc49-a23c-4dd9-af84-525dca03041d\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") "
Sep 09 21:47:53.842479 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.842356    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-metrics-certs\") pod \"fd69dc49-a23c-4dd9-af84-525dca03041d\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") "
Sep 09 21:47:53.842479 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.842385    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4nxn\" (UniqueName: \"kubernetes.io/projected/fd69dc49-a23c-4dd9-af84-525dca03041d-kube-api-access-z4nxn\") pod \"fd69dc49-a23c-4dd9-af84-525dca03041d\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") "
Sep 09 21:47:53.842479 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.842412    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls\") pod \"fd69dc49-a23c-4dd9-af84-525dca03041d\" (UID: \"fd69dc49-a23c-4dd9-af84-525dca03041d\") "
Sep 09 21:47:53.842670 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.842642    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd69dc49-a23c-4dd9-af84-525dca03041d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd69dc49-a23c-4dd9-af84-525dca03041d" (UID: "fd69dc49-a23c-4dd9-af84-525dca03041d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:47:53.844880 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.844852    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd69dc49-a23c-4dd9-af84-525dca03041d-kube-api-access-z4nxn" (OuterVolumeSpecName: "kube-api-access-z4nxn") pod "fd69dc49-a23c-4dd9-af84-525dca03041d" (UID: "fd69dc49-a23c-4dd9-af84-525dca03041d"). InnerVolumeSpecName "kube-api-access-z4nxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:53.845028 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.844901    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fd69dc49-a23c-4dd9-af84-525dca03041d" (UID: "fd69dc49-a23c-4dd9-af84-525dca03041d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:47:53.845028 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.844957    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd69dc49-a23c-4dd9-af84-525dca03041d" (UID: "fd69dc49-a23c-4dd9-af84-525dca03041d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:47:53.940325 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.940285    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:53.943463 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.943434    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd69dc49-a23c-4dd9-af84-525dca03041d-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:47:53.943463 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.943460    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:47:53.943631 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.943469    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nxn\" (UniqueName: \"kubernetes.io/projected/fd69dc49-a23c-4dd9-af84-525dca03041d-kube-api-access-z4nxn\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:47:53.943631 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:53.943478    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd69dc49-a23c-4dd9-af84-525dca03041d-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:47:54.066378 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.066293    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:54.148065 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.148039    2585 generic.go:356] "Generic (PLEG): container finished" podID="fd69dc49-a23c-4dd9-af84-525dca03041d" containerID="5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca" exitCode=0
Sep 09 21:47:54.148247 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.148087    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"
Sep 09 21:47:54.148247 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.148125    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5" event={"ID":"fd69dc49-a23c-4dd9-af84-525dca03041d","Type":"ContainerDied","Data":"5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca"}
Sep 09 21:47:54.148247 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.148181    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5" event={"ID":"fd69dc49-a23c-4dd9-af84-525dca03041d","Type":"ContainerDied","Data":"312fda78c7f6a87d0fe1cebe6eba2b874d0179d20e07e71b37a51cb8d4dcd276"}
Sep 09 21:47:54.148247 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.148228    2585 scope.go:117] "RemoveContainer" containerID="5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca"
Sep 09 21:47:54.157390 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.157362    2585 scope.go:117] "RemoveContainer" containerID="5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca"
Sep 09 21:47:54.157659 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:47:54.157639    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca\": container with ID starting with 5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca not found: ID does not exist" containerID="5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca"
Sep 09 21:47:54.157711 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.157667    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca"} err="failed to get container status \"5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca\": rpc error: code = NotFound desc = could not find container \"5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca\": container with ID starting with 5bcf27f513ee7b42041b35f634d582ab91db98419c9b3ad36af274d6c8eb28ca not found: ID does not exist"
Sep 09 21:47:54.179613 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.179584    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"]
Sep 09 21:47:54.186806 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:54.186786    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b27eb4af-76cc474494-trlm5"]
Sep 09 21:47:55.355562 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:55.355532    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:55.752304 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:55.752250    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:56.129336 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:56.129261    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd69dc49-a23c-4dd9-af84-525dca03041d" path="/var/lib/kubelet/pods/fd69dc49-a23c-4dd9-af84-525dca03041d/volumes"
Sep 09 21:47:56.138188 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:56.138145    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:56.489235 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:56.489180    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:58.342354 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:58.342317    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:58.487736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:58.487699    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:59.512198 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.512163    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"]
Sep 09 21:47:59.513327 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.513300    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd69dc49-a23c-4dd9-af84-525dca03041d" containerName="maas-api"
Sep 09 21:47:59.513327 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.513329    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd69dc49-a23c-4dd9-af84-525dca03041d" containerName="maas-api"
Sep 09 21:47:59.513489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.513475    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd69dc49-a23c-4dd9-af84-525dca03041d" containerName="maas-api"
Sep 09 21:47:59.517316 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.517294    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.519935 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.519914    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ef9b2234-serving-cert\""
Sep 09 21:47:59.532291 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.532267    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"]
Sep 09 21:47:59.596086 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.596057    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.596244 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.596098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-openshift-service-ca\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.596244 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.596131    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2tdr\" (UniqueName: \"kubernetes.io/projected/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-kube-api-access-h2tdr\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.596339 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.596237    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-metrics-certs\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.599246 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.599226    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:59.697682 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.697640    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.697860 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.697697    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-openshift-service-ca\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.697860 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.697750    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tdr\" (UniqueName: \"kubernetes.io/projected/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-kube-api-access-h2tdr\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.697860 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:47:59.697788    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-serving-cert: secret "maas-api-e2e-derive-ef9b2234-serving-cert" not found
Sep 09 21:47:59.698043 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.697796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-metrics-certs\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.698043 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:47:59.697862    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls podName:e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6 nodeName:}" failed. No retries permitted until 2026-09-09 21:48:00.197838578 +0000 UTC m=+882.572162380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls") pod "maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb" (UID: "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6") : secret "maas-api-e2e-derive-ef9b2234-serving-cert" not found
Sep 09 21:47:59.698433 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.698402    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-openshift-service-ca\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.700549 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.700530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-metrics-certs\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.702407 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.702386    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:59.706542 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.706520    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2tdr\" (UniqueName: \"kubernetes.io/projected/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-kube-api-access-h2tdr\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:47:59.978280 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:47:59.978246    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:00.203510 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.203473    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:48:00.205989 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.205965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls\") pod \"maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:48:00.246581 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.246510    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:00.332352 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.332322    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:00.430818 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.430785    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:48:00.507514 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.507428    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"]
Sep 09 21:48:00.511428 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.511409    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:00.524288 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.524188    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ef9b2234-dockercfg-jm2w9\""
Sep 09 21:48:00.525500 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.525448    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ef9b2234\""
Sep 09 21:48:00.528507 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.528447    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"]
Sep 09 21:48:00.608859 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.608831    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1374b3-1c05-4cdc-9127-ec00165c121a-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8\" (UID: \"8e1374b3-1c05-4cdc-9127-ec00165c121a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:00.609023 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.608879    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv26q\" (UniqueName: \"kubernetes.io/projected/8e1374b3-1c05-4cdc-9127-ec00165c121a-kube-api-access-pv26q\") pod \"payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8\" (UID: \"8e1374b3-1c05-4cdc-9127-ec00165c121a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:00.708261 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.708238    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"]
Sep 09 21:48:00.709943 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.709915    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1374b3-1c05-4cdc-9127-ec00165c121a-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8\" (UID: \"8e1374b3-1c05-4cdc-9127-ec00165c121a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:00.710069 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.709978    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv26q\" (UniqueName: \"kubernetes.io/projected/8e1374b3-1c05-4cdc-9127-ec00165c121a-kube-api-access-pv26q\") pod \"payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8\" (UID: \"8e1374b3-1c05-4cdc-9127-ec00165c121a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:00.710716 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.710690    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1374b3-1c05-4cdc-9127-ec00165c121a-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8\" (UID: \"8e1374b3-1c05-4cdc-9127-ec00165c121a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:00.710829 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:48:00.710696    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f9b52f_3d90_4fcb_ad8c_8ba0545b31c6.slice/crio-acab0ca3477f32de8e215cfc5a306b6ccc77a34599b44c840a783e3525bd7dd8 WatchSource:0}: Error finding container acab0ca3477f32de8e215cfc5a306b6ccc77a34599b44c840a783e3525bd7dd8: Status 404 returned error can't find the container with id acab0ca3477f32de8e215cfc5a306b6ccc77a34599b44c840a783e3525bd7dd8
Sep 09 21:48:00.718962 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.718939    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv26q\" (UniqueName: \"kubernetes.io/projected/8e1374b3-1c05-4cdc-9127-ec00165c121a-kube-api-access-pv26q\") pod \"payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8\" (UID: \"8e1374b3-1c05-4cdc-9127-ec00165c121a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:00.832135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:00.832048    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:01.118157 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:01.118082    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"]
Sep 09 21:48:01.121546 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:48:01.121515    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1374b3_1c05_4cdc_9127_ec00165c121a.slice/crio-3ecbf1b970b34def9de4c4a64477bcd6ef157d6fcee13340572ef9184534ac23 WatchSource:0}: Error finding container 3ecbf1b970b34def9de4c4a64477bcd6ef157d6fcee13340572ef9184534ac23: Status 404 returned error can't find the container with id 3ecbf1b970b34def9de4c4a64477bcd6ef157d6fcee13340572ef9184534ac23
Sep 09 21:48:01.195680 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:01.195650    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8" event={"ID":"8e1374b3-1c05-4cdc-9127-ec00165c121a","Type":"ContainerStarted","Data":"3ecbf1b970b34def9de4c4a64477bcd6ef157d6fcee13340572ef9184534ac23"}
Sep 09 21:48:01.195792 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:01.195761    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:01.197083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:01.197062    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb" event={"ID":"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6","Type":"ContainerStarted","Data":"acab0ca3477f32de8e215cfc5a306b6ccc77a34599b44c840a783e3525bd7dd8"}
Sep 09 21:48:01.218231 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:01.218171    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8" podStartSLOduration=1.218156402 podStartE2EDuration="1.218156402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:01.216092525 +0000 UTC m=+883.590416343" watchObservedRunningTime="2026-09-09 21:48:01.218156402 +0000 UTC m=+883.592480469"
Sep 09 21:48:02.203358 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:02.203324    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8" event={"ID":"8e1374b3-1c05-4cdc-9127-ec00165c121a","Type":"ContainerStarted","Data":"17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303"}
Sep 09 21:48:02.204986 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:02.204962    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb" event={"ID":"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6","Type":"ContainerStarted","Data":"1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996"}
Sep 09 21:48:02.205104 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:02.205075    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:48:02.231040 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:02.230984    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb" podStartSLOduration=2.7087282889999997 podStartE2EDuration="3.230968204s" totalImagesPullingTime="522.239915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:02.229056567 +0000 UTC m=+884.603380392" watchObservedRunningTime="2026-09-09 21:48:02.230968204 +0000 UTC m=+884.605292068"
Sep 09 21:48:08.215929 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:08.215893    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:48:13.213669 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:13.213588    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:16.202993 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:16.202962    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:18.052090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:18.052062    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:48:18.056237 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:18.056199    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:48:25.397987 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:25.397956    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:29.800523 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:29.800481    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:30.438825 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.438792    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"]
Sep 09 21:48:30.439306 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.439272    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb" podUID="e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" containerName="maas-api" containerID="cri-o://1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996" gracePeriod=30
Sep 09 21:48:30.475277 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.475245    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:30.496307 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.496270    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"]
Sep 09 21:48:30.496579 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.496552    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8" podUID="8e1374b3-1c05-4cdc-9127-ec00165c121a" containerName="payload-processing" containerID="cri-o://17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303" gracePeriod=30
Sep 09 21:48:30.575630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.575592    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:30.674148 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.674063    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:30.749075 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.748754    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:48:30.782242 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.777403    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:30.870928 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.870890    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:30.883858 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.883833    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2tdr\" (UniqueName: \"kubernetes.io/projected/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-kube-api-access-h2tdr\") pod \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") "
Sep 09 21:48:30.883984 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.883876    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv26q\" (UniqueName: \"kubernetes.io/projected/8e1374b3-1c05-4cdc-9127-ec00165c121a-kube-api-access-pv26q\") pod \"8e1374b3-1c05-4cdc-9127-ec00165c121a\" (UID: \"8e1374b3-1c05-4cdc-9127-ec00165c121a\") "
Sep 09 21:48:30.883984 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.883924    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1374b3-1c05-4cdc-9127-ec00165c121a-plugins-config-volume\") pod \"8e1374b3-1c05-4cdc-9127-ec00165c121a\" (UID: \"8e1374b3-1c05-4cdc-9127-ec00165c121a\") "
Sep 09 21:48:30.884086 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.883986    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls\") pod \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") "
Sep 09 21:48:30.884086 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.884034    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-metrics-certs\") pod \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") "
Sep 09 21:48:30.884198 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.884127    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-openshift-service-ca\") pod \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\" (UID: \"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6\") "
Sep 09 21:48:30.884459 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.884426    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1374b3-1c05-4cdc-9127-ec00165c121a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e1374b3-1c05-4cdc-9127-ec00165c121a" (UID: "8e1374b3-1c05-4cdc-9127-ec00165c121a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:30.884746 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.884606    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1374b3-1c05-4cdc-9127-ec00165c121a-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:30.884977 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.884952    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" (UID: "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:30.887109 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.887088    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1374b3-1c05-4cdc-9127-ec00165c121a-kube-api-access-pv26q" (OuterVolumeSpecName: "kube-api-access-pv26q") pod "8e1374b3-1c05-4cdc-9127-ec00165c121a" (UID: "8e1374b3-1c05-4cdc-9127-ec00165c121a"). InnerVolumeSpecName "kube-api-access-pv26q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:30.888410 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.888381    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-kube-api-access-h2tdr" (OuterVolumeSpecName: "kube-api-access-h2tdr") pod "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" (UID: "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6"). InnerVolumeSpecName "kube-api-access-h2tdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:30.888503 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.888404    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" (UID: "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:30.888503 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.888479    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" (UID: "e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:30.978508 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.978423    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:30.985724 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.985703    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:30.985784 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.985728    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:30.985784 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.985738    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2tdr\" (UniqueName: \"kubernetes.io/projected/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-kube-api-access-h2tdr\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:30.985784 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.985747    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv26q\" (UniqueName: \"kubernetes.io/projected/8e1374b3-1c05-4cdc-9127-ec00165c121a-kube-api-access-pv26q\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:30.985784 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:30.985761    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:31.357444 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.357339    2585 generic.go:356] "Generic (PLEG): container finished" podID="8e1374b3-1c05-4cdc-9127-ec00165c121a" containerID="17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303" exitCode=0
Sep 09 21:48:31.357444 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.357379    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"
Sep 09 21:48:31.357444 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.357439    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8" event={"ID":"8e1374b3-1c05-4cdc-9127-ec00165c121a","Type":"ContainerDied","Data":"17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303"}
Sep 09 21:48:31.357761 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.357487    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8" event={"ID":"8e1374b3-1c05-4cdc-9127-ec00165c121a","Type":"ContainerDied","Data":"3ecbf1b970b34def9de4c4a64477bcd6ef157d6fcee13340572ef9184534ac23"}
Sep 09 21:48:31.357761 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.357509    2585 scope.go:117] "RemoveContainer" containerID="17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303"
Sep 09 21:48:31.358850 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.358830    2585 generic.go:356] "Generic (PLEG): container finished" podID="e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" containerID="1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996" exitCode=0
Sep 09 21:48:31.358927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.358882    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"
Sep 09 21:48:31.358927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.358911    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb" event={"ID":"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6","Type":"ContainerDied","Data":"1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996"}
Sep 09 21:48:31.359045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.358952    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb" event={"ID":"e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6","Type":"ContainerDied","Data":"acab0ca3477f32de8e215cfc5a306b6ccc77a34599b44c840a783e3525bd7dd8"}
Sep 09 21:48:31.368338 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.368319    2585 scope.go:117] "RemoveContainer" containerID="17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303"
Sep 09 21:48:31.368644 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:48:31.368628    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303\": container with ID starting with 17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303 not found: ID does not exist" containerID="17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303"
Sep 09 21:48:31.368700 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.368653    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303"} err="failed to get container status \"17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303\": rpc error: code = NotFound desc = could not find container \"17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303\": container with ID starting with 17869b0066df821863a500fea20fafcb6fbb3065d71ba71ba036ccfb50fa6303 not found: ID does not exist"
Sep 09 21:48:31.368700 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.368667    2585 scope.go:117] "RemoveContainer" containerID="1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996"
Sep 09 21:48:31.376925 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.376910    2585 scope.go:117] "RemoveContainer" containerID="1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996"
Sep 09 21:48:31.377184 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:48:31.377166    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996\": container with ID starting with 1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996 not found: ID does not exist" containerID="1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996"
Sep 09 21:48:31.377264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.377192    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996"} err="failed to get container status \"1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996\": rpc error: code = NotFound desc = could not find container \"1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996\": container with ID starting with 1317e270990bad10daadcc1ee09d78f03cf03066e285d20d7f8a44e0b0680996 not found: ID does not exist"
Sep 09 21:48:31.380056 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.380035    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"]
Sep 09 21:48:31.386220 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.386184    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef9b2234-6d4b9747c7-q6hq8"]
Sep 09 21:48:31.395522 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.395502    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"]
Sep 09 21:48:31.399250 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.399227    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef9b2234-965b6b5b9-8rdsb"]
Sep 09 21:48:31.497003 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.496974    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:31.591954 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.591920    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:31.691020 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:31.690968    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:32.127892 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:32.127816    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1374b3-1c05-4cdc-9127-ec00165c121a" path="/var/lib/kubelet/pods/8e1374b3-1c05-4cdc-9127-ec00165c121a/volumes"
Sep 09 21:48:32.128257 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:32.128190    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" path="/var/lib/kubelet/pods/e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6/volumes"
Sep 09 21:48:33.015772 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:33.015739    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:35.490924 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:35.490881    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:36.019264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.019229    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"]
Sep 09 21:48:36.019536 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.019508    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz" podUID="3f2be80d-36cc-4489-8eb7-a77246c44f5c" containerName="maas-api" containerID="cri-o://a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345" gracePeriod=30
Sep 09 21:48:36.044344 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.044314    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:36.071151 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.071119    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"]
Sep 09 21:48:36.071418 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.071394    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs" podUID="990459a4-88aa-4333-81c1-83f16df14b27" containerName="payload-processing" containerID="cri-o://ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483" gracePeriod=30
Sep 09 21:48:36.076415 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.076324    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"]
Sep 09 21:48:36.076591 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.076552    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn" podUID="899bd60a-dd42-4663-a771-0d44e4ebc153" containerName="payload-pre-processing" containerID="cri-o://2f9670a07e397493dd42ad9c011f0ed31167d06335d6ebdd6998b85f568ae57a" gracePeriod=30
Sep 09 21:48:36.115337 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.115306    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:36.176704 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.176667    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:36.241792 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.241745    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:36.344056 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.344031    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:48:36.383819 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.383777    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:48:36.388353 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.388324    2585 generic.go:356] "Generic (PLEG): container finished" podID="3f2be80d-36cc-4489-8eb7-a77246c44f5c" containerID="a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345" exitCode=0
Sep 09 21:48:36.388495 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.388363    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"
Sep 09 21:48:36.388495 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.388403    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz" event={"ID":"3f2be80d-36cc-4489-8eb7-a77246c44f5c","Type":"ContainerDied","Data":"a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345"}
Sep 09 21:48:36.388495 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.388449    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz" event={"ID":"3f2be80d-36cc-4489-8eb7-a77246c44f5c","Type":"ContainerDied","Data":"35932fa3dc24afe89e8a99d5af4c54efed786876387ce019af5fb2a248cd5893"}
Sep 09 21:48:36.388495 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.388482    2585 scope.go:117] "RemoveContainer" containerID="a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345"
Sep 09 21:48:36.390572 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.390547    2585 generic.go:356] "Generic (PLEG): container finished" podID="899bd60a-dd42-4663-a771-0d44e4ebc153" containerID="2f9670a07e397493dd42ad9c011f0ed31167d06335d6ebdd6998b85f568ae57a" exitCode=0
Sep 09 21:48:36.390677 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.390649    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn" event={"ID":"899bd60a-dd42-4663-a771-0d44e4ebc153","Type":"ContainerDied","Data":"2f9670a07e397493dd42ad9c011f0ed31167d06335d6ebdd6998b85f568ae57a"}
Sep 09 21:48:36.392126 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.392105    2585 generic.go:356] "Generic (PLEG): container finished" podID="990459a4-88aa-4333-81c1-83f16df14b27" containerID="ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483" exitCode=0
Sep 09 21:48:36.392270 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.392171    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs" event={"ID":"990459a4-88aa-4333-81c1-83f16df14b27","Type":"ContainerDied","Data":"ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483"}
Sep 09 21:48:36.392270 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.392199    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs" event={"ID":"990459a4-88aa-4333-81c1-83f16df14b27","Type":"ContainerDied","Data":"4a2e7abe804584d0c1c75032d05c72458c0b0d7f629186111e776b73733901c1"}
Sep 09 21:48:36.392270 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.392271    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"
Sep 09 21:48:36.397780 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.397759    2585 scope.go:117] "RemoveContainer" containerID="a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345"
Sep 09 21:48:36.398066 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:48:36.398043    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345\": container with ID starting with a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345 not found: ID does not exist" containerID="a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345"
Sep 09 21:48:36.398134 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.398079    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345"} err="failed to get container status \"a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345\": rpc error: code = NotFound desc = could not find container \"a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345\": container with ID starting with a3d91c616e797b2c0b8dea5bd947c2bfa58087e9491f546cd49fe777b9508345 not found: ID does not exist"
Sep 09 21:48:36.398134 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.398101    2585 scope.go:117] "RemoveContainer" containerID="ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483"
Sep 09 21:48:36.408639 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.408616    2585 scope.go:117] "RemoveContainer" containerID="ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483"
Sep 09 21:48:36.408976 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:48:36.408955    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483\": container with ID starting with ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483 not found: ID does not exist" containerID="ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483"
Sep 09 21:48:36.409078 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.408986    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483"} err="failed to get container status \"ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483\": rpc error: code = NotFound desc = could not find container \"ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483\": container with ID starting with ba611f79bc04127d54b1a12c7d286fe6a753eb00b34d50b1fe595060f4a69483 not found: ID does not exist"
Sep 09 21:48:36.420777 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.420749    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:48:36.537495 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537461    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899bd60a-dd42-4663-a771-0d44e4ebc153-plugins-config-volume\") pod \"899bd60a-dd42-4663-a771-0d44e4ebc153\" (UID: \"899bd60a-dd42-4663-a771-0d44e4ebc153\") "
Sep 09 21:48:36.537951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537564    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqd9c\" (UniqueName: \"kubernetes.io/projected/990459a4-88aa-4333-81c1-83f16df14b27-kube-api-access-dqd9c\") pod \"990459a4-88aa-4333-81c1-83f16df14b27\" (UID: \"990459a4-88aa-4333-81c1-83f16df14b27\") "
Sep 09 21:48:36.537951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537599    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2be80d-36cc-4489-8eb7-a77246c44f5c-openshift-service-ca\") pod \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") "
Sep 09 21:48:36.537951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537655    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn5ql\" (UniqueName: \"kubernetes.io/projected/3f2be80d-36cc-4489-8eb7-a77246c44f5c-kube-api-access-wn5ql\") pod \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") "
Sep 09 21:48:36.537951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537679    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990459a4-88aa-4333-81c1-83f16df14b27-plugins-config-volume\") pod \"990459a4-88aa-4333-81c1-83f16df14b27\" (UID: \"990459a4-88aa-4333-81c1-83f16df14b27\") "
Sep 09 21:48:36.537951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537706    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-maas-api-tls\") pod \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") "
Sep 09 21:48:36.537951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537754    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-metrics-certs\") pod \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\" (UID: \"3f2be80d-36cc-4489-8eb7-a77246c44f5c\") "
Sep 09 21:48:36.537951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537786    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmshl\" (UniqueName: \"kubernetes.io/projected/899bd60a-dd42-4663-a771-0d44e4ebc153-kube-api-access-gmshl\") pod \"899bd60a-dd42-4663-a771-0d44e4ebc153\" (UID: \"899bd60a-dd42-4663-a771-0d44e4ebc153\") "
Sep 09 21:48:36.537951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.537785    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899bd60a-dd42-4663-a771-0d44e4ebc153-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "899bd60a-dd42-4663-a771-0d44e4ebc153" (UID: "899bd60a-dd42-4663-a771-0d44e4ebc153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:36.538380 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.538084    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899bd60a-dd42-4663-a771-0d44e4ebc153-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.538380 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.538077    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990459a4-88aa-4333-81c1-83f16df14b27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "990459a4-88aa-4333-81c1-83f16df14b27" (UID: "990459a4-88aa-4333-81c1-83f16df14b27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:36.538529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.538486    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2be80d-36cc-4489-8eb7-a77246c44f5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f2be80d-36cc-4489-8eb7-a77246c44f5c" (UID: "3f2be80d-36cc-4489-8eb7-a77246c44f5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:36.540021 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.539989    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990459a4-88aa-4333-81c1-83f16df14b27-kube-api-access-dqd9c" (OuterVolumeSpecName: "kube-api-access-dqd9c") pod "990459a4-88aa-4333-81c1-83f16df14b27" (UID: "990459a4-88aa-4333-81c1-83f16df14b27"). InnerVolumeSpecName "kube-api-access-dqd9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:36.540147 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.540035    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f2be80d-36cc-4489-8eb7-a77246c44f5c" (UID: "3f2be80d-36cc-4489-8eb7-a77246c44f5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:36.540191 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.540148    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f2be80d-36cc-4489-8eb7-a77246c44f5c" (UID: "3f2be80d-36cc-4489-8eb7-a77246c44f5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:36.540456 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.540439    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2be80d-36cc-4489-8eb7-a77246c44f5c-kube-api-access-wn5ql" (OuterVolumeSpecName: "kube-api-access-wn5ql") pod "3f2be80d-36cc-4489-8eb7-a77246c44f5c" (UID: "3f2be80d-36cc-4489-8eb7-a77246c44f5c"). InnerVolumeSpecName "kube-api-access-wn5ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:36.540517 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.540493    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899bd60a-dd42-4663-a771-0d44e4ebc153-kube-api-access-gmshl" (OuterVolumeSpecName: "kube-api-access-gmshl") pod "899bd60a-dd42-4663-a771-0d44e4ebc153" (UID: "899bd60a-dd42-4663-a771-0d44e4ebc153"). InnerVolumeSpecName "kube-api-access-gmshl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:36.564083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.564056    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:36.639544 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.639464    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqd9c\" (UniqueName: \"kubernetes.io/projected/990459a4-88aa-4333-81c1-83f16df14b27-kube-api-access-dqd9c\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.639544 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.639503    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2be80d-36cc-4489-8eb7-a77246c44f5c-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.639544 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.639518    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5ql\" (UniqueName: \"kubernetes.io/projected/3f2be80d-36cc-4489-8eb7-a77246c44f5c-kube-api-access-wn5ql\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.639544 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.639530    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990459a4-88aa-4333-81c1-83f16df14b27-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.639544 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.639543    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.639780 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.639555    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2be80d-36cc-4489-8eb7-a77246c44f5c-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.639780 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.639567    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmshl\" (UniqueName: \"kubernetes.io/projected/899bd60a-dd42-4663-a771-0d44e4ebc153-kube-api-access-gmshl\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.710630 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.710594    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"]
Sep 09 21:48:36.712383 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.712360    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66cedd-764d4fcdbd-hwswz"]
Sep 09 21:48:36.726283 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.726260    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"]
Sep 09 21:48:36.730844 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:36.730826    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66cedd-7676d8b945-jrxxs"]
Sep 09 21:48:37.400720 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:37.400636    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn" event={"ID":"899bd60a-dd42-4663-a771-0d44e4ebc153","Type":"ContainerDied","Data":"c8668518f2ee9178a0cde1642dbf7f29226cd02d458f70414db4463a547252f4"}
Sep 09 21:48:37.400720 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:37.400681    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"
Sep 09 21:48:37.400720 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:37.400707    2585 scope.go:117] "RemoveContainer" containerID="2f9670a07e397493dd42ad9c011f0ed31167d06335d6ebdd6998b85f568ae57a"
Sep 09 21:48:37.424098 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:37.424075    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"]
Sep 09 21:48:37.427514 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:37.427490    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66cedd-74bccd9db-bhprn"]
Sep 09 21:48:38.135388 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:38.135354    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2be80d-36cc-4489-8eb7-a77246c44f5c" path="/var/lib/kubelet/pods/3f2be80d-36cc-4489-8eb7-a77246c44f5c/volumes"
Sep 09 21:48:38.135751 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:38.135700    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899bd60a-dd42-4663-a771-0d44e4ebc153" path="/var/lib/kubelet/pods/899bd60a-dd42-4663-a771-0d44e4ebc153/volumes"
Sep 09 21:48:38.135997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:38.135984    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="990459a4-88aa-4333-81c1-83f16df14b27" path="/var/lib/kubelet/pods/990459a4-88aa-4333-81c1-83f16df14b27/volumes"
Sep 09 21:48:38.420246 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:38.420198    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:38.861969 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:38.861886    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:38.961034 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:38.960994    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:39.061577 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:39.061545    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:39.161370 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:39.161342    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:39.509284 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:39.509188    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:41.646416 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:48:41.646387    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:00.789016 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.788978    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"]
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789404    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2be80d-36cc-4489-8eb7-a77246c44f5c" containerName="maas-api"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789419    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2be80d-36cc-4489-8eb7-a77246c44f5c" containerName="maas-api"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789431    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1374b3-1c05-4cdc-9127-ec00165c121a" containerName="payload-processing"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789446    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1374b3-1c05-4cdc-9127-ec00165c121a" containerName="payload-processing"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789459    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990459a4-88aa-4333-81c1-83f16df14b27" containerName="payload-processing"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789465    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="990459a4-88aa-4333-81c1-83f16df14b27" containerName="payload-processing"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789473    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" containerName="maas-api"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789480    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" containerName="maas-api"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789493    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899bd60a-dd42-4663-a771-0d44e4ebc153" containerName="payload-pre-processing"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789502    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="899bd60a-dd42-4663-a771-0d44e4ebc153" containerName="payload-pre-processing"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789568    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="899bd60a-dd42-4663-a771-0d44e4ebc153" containerName="payload-pre-processing"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789582    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f9b52f-3d90-4fcb-ad8c-8ba0545b31c6" containerName="maas-api"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789592    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2be80d-36cc-4489-8eb7-a77246c44f5c" containerName="maas-api"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789598    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="990459a4-88aa-4333-81c1-83f16df14b27" containerName="payload-processing"
Sep 09 21:49:00.789697 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.789604    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1374b3-1c05-4cdc-9127-ec00165c121a" containerName="payload-processing"
Sep 09 21:49:00.793076 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.793056    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.796424 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.796387    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"default-dockercfg-h6wml\""
Sep 09 21:49:00.796424 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.796408    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"premium-w4-3fa986-kserve-self-signed-certs\""
Sep 09 21:49:00.796617 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.796462    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"openshift-service-ca.crt\""
Sep 09 21:49:00.796617 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.796387    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"kube-root-ca.crt\""
Sep 09 21:49:00.802661 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.802641    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"]
Sep 09 21:49:00.826085 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.826060    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:00.853580 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.853556    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tls-certs\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.853682 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.853599    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-home\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.853682 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.853662    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-dshm\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.853758 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.853731    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tn75\" (UniqueName: \"kubernetes.io/projected/6ab68ead-a5d8-4271-9e66-6723e3b926fe-kube-api-access-4tn75\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.853758 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.853753    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tmp-dir\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.853822 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.853776    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-model-cache\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.954374 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.954335    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tmp-dir\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.954374 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.954375    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-model-cache\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.954588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.954400    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tls-certs\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.954588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.954431    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-home\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.954588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.954479    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-dshm\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.954588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.954507    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tn75\" (UniqueName: \"kubernetes.io/projected/6ab68ead-a5d8-4271-9e66-6723e3b926fe-kube-api-access-4tn75\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.954835 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.954809    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-model-cache\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.954926 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.954884    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-home\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.955427 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.955406    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tmp-dir\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.956874 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.956851    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-dshm\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.957406 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.957389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tls-certs\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:00.965740 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:00.965715    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tn75\" (UniqueName: \"kubernetes.io/projected/6ab68ead-a5d8-4271-9e66-6723e3b926fe-kube-api-access-4tn75\") pod \"premium-w4-3fa986-kserve-f6c6754cf-zgckf\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") " pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:01.103259 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:01.103157    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:01.379917 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:01.379890    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"]
Sep 09 21:49:01.381721 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:49:01.381693    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ab68ead_a5d8_4271_9e66_6723e3b926fe.slice/crio-dda5c886890953ca8fddc94347cd53c380948cf200cb58f98e95734025568fd0 WatchSource:0}: Error finding container dda5c886890953ca8fddc94347cd53c380948cf200cb58f98e95734025568fd0: Status 404 returned error can't find the container with id dda5c886890953ca8fddc94347cd53c380948cf200cb58f98e95734025568fd0
Sep 09 21:49:01.522703 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:01.522664    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf" event={"ID":"6ab68ead-a5d8-4271-9e66-6723e3b926fe","Type":"ContainerStarted","Data":"a1fd68d284db4ea83f4548e93e18bb549e11a77dfbffd7c9baf12d5987a02d23"}
Sep 09 21:49:01.522703 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:01.522714    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf" event={"ID":"6ab68ead-a5d8-4271-9e66-6723e3b926fe","Type":"ContainerStarted","Data":"dda5c886890953ca8fddc94347cd53c380948cf200cb58f98e95734025568fd0"}
Sep 09 21:49:01.543144 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:01.543090    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf" podStartSLOduration=1.543074901 podStartE2EDuration="1.543074901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:01.540687052 +0000 UTC m=+943.915010879" watchObservedRunningTime="2026-09-09 21:49:01.543074901 +0000 UTC m=+943.917398747"
Sep 09 21:49:02.660585 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:02.660533    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:05.371360 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:05.371324    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:11.061467 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:11.061419    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:11.103396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:11.103365    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:11.103575 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:11.103418    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:11.116672 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:11.116642    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:11.150241 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:11.150192    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:11.222102 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:11.222070    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:11.590588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:11.590556    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 21:49:18.652165 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:18.652125    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:18.906683 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:18.906603    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:19.340676 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:19.340584    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:19.619388 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:19.619274    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:20.335761 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:20.335724    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:22.291460 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:22.291427    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:22.393474 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:22.393427    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:23.485492 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.485464    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"]
Sep 09 21:49:23.490462 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.490436    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.493231 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.493184    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-331c9d-serving-cert\""
Sep 09 21:49:23.504079 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.504057    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"]
Sep 09 21:49:23.531249 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.531192    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:23.564621 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.564589    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-metrics-certs\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.564801 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.564686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.564801 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.564729    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztm7\" (UniqueName: \"kubernetes.io/projected/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-kube-api-access-2ztm7\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.564801 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.564774    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-openshift-service-ca\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.632164 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.632124    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:23.665771 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.665743    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-metrics-certs\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.665949 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.665823    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.665949 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.665851    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztm7\" (UniqueName: \"kubernetes.io/projected/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-kube-api-access-2ztm7\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.665949 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.665883    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-openshift-service-ca\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.666077 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:49:23.665969    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-serving-cert: secret "maas-api-e2e-explicit-331c9d-serving-cert" not found
Sep 09 21:49:23.666077 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:49:23.666053    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls podName:2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f nodeName:}" failed. No retries permitted until 2026-09-09 21:49:24.166032069 +0000 UTC m=+966.540355877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls") pod "maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk" (UID: "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f") : secret "maas-api-e2e-explicit-331c9d-serving-cert" not found
Sep 09 21:49:23.666624 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.666557    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-openshift-service-ca\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.668388 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.668368    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-metrics-certs\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.681083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.681059    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztm7\" (UniqueName: \"kubernetes.io/projected/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-kube-api-access-2ztm7\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:23.915253 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:23.915196    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:24.147672 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.147638    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:24.170907 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.170824    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:24.173835 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.173811    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls\") pod \"maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:24.247094 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.247054    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:24.286984 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.286950    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"]
Sep 09 21:49:24.290816 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.290799    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:24.292935 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.292910    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-331c9d-dockercfg-z9tmg\""
Sep 09 21:49:24.293446 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.293419    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-331c9d\""
Sep 09 21:49:24.301979 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.301958    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"]
Sep 09 21:49:24.336954 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.336901    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"]
Sep 09 21:49:24.346983 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.346440    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:24.346983 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.346826    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"]
Sep 09 21:49:24.372082 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.372053    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnffz\" (UniqueName: \"kubernetes.io/projected/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-kube-api-access-dnffz\") pod \"payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5\" (UID: \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\") " pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:24.372339 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.372125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7421895d-bbd4-4d48-a22d-7258bb6beb2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2\" (UID: \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:24.372339 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.372148    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khmc\" (UniqueName: \"kubernetes.io/projected/7421895d-bbd4-4d48-a22d-7258bb6beb2f-kube-api-access-8khmc\") pod \"payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2\" (UID: \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:24.372339 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.372168    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5\" (UID: \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\") " pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:24.401893 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.401861    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:24.475458 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.473737    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnffz\" (UniqueName: \"kubernetes.io/projected/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-kube-api-access-dnffz\") pod \"payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5\" (UID: \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\") " pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:24.475458 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.473901    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7421895d-bbd4-4d48-a22d-7258bb6beb2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2\" (UID: \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:24.475458 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.473941    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khmc\" (UniqueName: \"kubernetes.io/projected/7421895d-bbd4-4d48-a22d-7258bb6beb2f-kube-api-access-8khmc\") pod \"payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2\" (UID: \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:24.475458 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.473975    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5\" (UID: \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\") " pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:24.475458 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.475093    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7421895d-bbd4-4d48-a22d-7258bb6beb2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2\" (UID: \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:24.475458 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.475391    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5\" (UID: \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\") " pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:24.483151 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.482911    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnffz\" (UniqueName: \"kubernetes.io/projected/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-kube-api-access-dnffz\") pod \"payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5\" (UID: \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\") " pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:24.484486 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.484464    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khmc\" (UniqueName: \"kubernetes.io/projected/7421895d-bbd4-4d48-a22d-7258bb6beb2f-kube-api-access-8khmc\") pod \"payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2\" (UID: \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:24.603762 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.603731    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:24.668449 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.667177    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:24.692183 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.691955    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"]
Sep 09 21:49:24.696040 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:49:24.695786    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2713ab25_b8c3_4b8f_bae2_fd86f3cd3e1f.slice/crio-b84481e82b986c9b6b81bc5f2b7eed70e80c9590c8064bf402d50052f490f606 WatchSource:0}: Error finding container b84481e82b986c9b6b81bc5f2b7eed70e80c9590c8064bf402d50052f490f606: Status 404 returned error can't find the container with id b84481e82b986c9b6b81bc5f2b7eed70e80c9590c8064bf402d50052f490f606
Sep 09 21:49:24.698147 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.698127    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:49:24.884171 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.884146    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"]
Sep 09 21:49:24.886600 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:49:24.886574    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f72e6a_21c8_4f97_b399_c798fb24f7f7.slice/crio-1ca676a2ad1dac6060a77c1565ec1880a21ecbdaa68d3fdcf40c48a6cb70dea6 WatchSource:0}: Error finding container 1ca676a2ad1dac6060a77c1565ec1880a21ecbdaa68d3fdcf40c48a6cb70dea6: Status 404 returned error can't find the container with id 1ca676a2ad1dac6060a77c1565ec1880a21ecbdaa68d3fdcf40c48a6cb70dea6
Sep 09 21:49:24.955373 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:24.955343    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"]
Sep 09 21:49:24.957190 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:49:24.957154    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7421895d_bbd4_4d48_a22d_7258bb6beb2f.slice/crio-08207e97a0f089d3ef53b4e3cdf997e48e639def59fa707ebd5ef866cfbf6f25 WatchSource:0}: Error finding container 08207e97a0f089d3ef53b4e3cdf997e48e639def59fa707ebd5ef866cfbf6f25: Status 404 returned error can't find the container with id 08207e97a0f089d3ef53b4e3cdf997e48e639def59fa707ebd5ef866cfbf6f25
Sep 09 21:49:25.653811 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.653765    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2" event={"ID":"7421895d-bbd4-4d48-a22d-7258bb6beb2f","Type":"ContainerStarted","Data":"2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb"}
Sep 09 21:49:25.654303 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.653899    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:25.654303 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.653943    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2" event={"ID":"7421895d-bbd4-4d48-a22d-7258bb6beb2f","Type":"ContainerStarted","Data":"08207e97a0f089d3ef53b4e3cdf997e48e639def59fa707ebd5ef866cfbf6f25"}
Sep 09 21:49:25.655953 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.655921    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5" event={"ID":"b2f72e6a-21c8-4f97-b399-c798fb24f7f7","Type":"ContainerStarted","Data":"ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f"}
Sep 09 21:49:25.656090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.655975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5" event={"ID":"b2f72e6a-21c8-4f97-b399-c798fb24f7f7","Type":"ContainerStarted","Data":"1ca676a2ad1dac6060a77c1565ec1880a21ecbdaa68d3fdcf40c48a6cb70dea6"}
Sep 09 21:49:25.656090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.656010    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:25.657940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.657917    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk" event={"ID":"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f","Type":"ContainerStarted","Data":"50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a"}
Sep 09 21:49:25.658074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.657960    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk" event={"ID":"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f","Type":"ContainerStarted","Data":"b84481e82b986c9b6b81bc5f2b7eed70e80c9590c8064bf402d50052f490f606"}
Sep 09 21:49:25.658074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.658047    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:25.672410 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.672354    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2" podStartSLOduration=1.6723357559999998 podStartE2EDuration="1.672335756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:25.670950202 +0000 UTC m=+968.045274033" watchObservedRunningTime="2026-09-09 21:49:25.672335756 +0000 UTC m=+968.046659580"
Sep 09 21:49:25.687175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.687127    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5" podStartSLOduration=1.687106014 podStartE2EDuration="1.687106014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:25.685869252 +0000 UTC m=+968.060193080" watchObservedRunningTime="2026-09-09 21:49:25.687106014 +0000 UTC m=+968.061429840"
Sep 09 21:49:25.705221 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:25.705163    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk" podStartSLOduration=2.294010531 podStartE2EDuration="2.705149772s" totalImagesPullingTime="411.139241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:25.703892674 +0000 UTC m=+968.078216499" watchObservedRunningTime="2026-09-09 21:49:25.705149772 +0000 UTC m=+968.079473596"
Sep 09 21:49:27.033890 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:27.033854    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:31.675099 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:31.675071    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:49:33.312487 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:33.312443    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:33.434714 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:33.434682    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:33.569760 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:33.569658    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:36.665539 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:36.665502    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:49:36.665967 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:36.665690    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:49:42.849911 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:42.849827    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:43.997948 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:43.997909    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:44.758767 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:44.758720    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:47.217674 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:47.217642    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:51.177097 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:51.177059    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:51.287185 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:51.287143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:51.375726 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:51.375692    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:51.475269 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:51.475111    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:52.073634 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:52.073599    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:52.175105 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:52.175074    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:52.272133 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:52.272101    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:53.727089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:53.727061    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:59.512865 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:59.512830    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:59.617803 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:49:59.617771    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:00.209504 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:00.209473    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:06.284762 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:06.284727    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:06.389152 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:06.389103    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:06.612017 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:06.611914    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:06.831308 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:06.831267    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:06.976172 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:06.976141    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:12.983917 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:12.983875    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:13.078765 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:13.078729    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:13.801027 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:13.800988    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:13.917818 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:13.917784    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:14.068191 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.068107    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:14.338126 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.338002    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:14.435526 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.435497    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:14.441353 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.441327    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"]
Sep 09 21:50:14.445065 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.445047    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:14.447374 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.447357    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-65d42a05-dockercfg-bcxdx\""
Sep 09 21:50:14.447458 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.447372    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-65d42a05\""
Sep 09 21:50:14.455833 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.455800    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"]
Sep 09 21:50:14.475735 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.475701    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"]
Sep 09 21:50:14.479293 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.479265    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:14.490250 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.490228    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"]
Sep 09 21:50:14.537022 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.536978    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84563778-dd9f-4173-bd6b-6a2744fbb067-plugins-config-volume\") pod \"payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh\" (UID: \"84563778-dd9f-4173-bd6b-6a2744fbb067\") " pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:14.537022 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.537021    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdh4\" (UniqueName: \"kubernetes.io/projected/84563778-dd9f-4173-bd6b-6a2744fbb067-kube-api-access-pbdh4\") pod \"payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh\" (UID: \"84563778-dd9f-4173-bd6b-6a2744fbb067\") " pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:14.537319 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.537090    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72c5x\" (UniqueName: \"kubernetes.io/projected/f2260124-5706-4fbe-865e-faf9ebe783a7-kube-api-access-72c5x\") pod \"payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5\" (UID: \"f2260124-5706-4fbe-865e-faf9ebe783a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:14.537319 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.537130    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2260124-5706-4fbe-865e-faf9ebe783a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5\" (UID: \"f2260124-5706-4fbe-865e-faf9ebe783a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:14.638375 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.638280    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84563778-dd9f-4173-bd6b-6a2744fbb067-plugins-config-volume\") pod \"payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh\" (UID: \"84563778-dd9f-4173-bd6b-6a2744fbb067\") " pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:14.638375 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.638332    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdh4\" (UniqueName: \"kubernetes.io/projected/84563778-dd9f-4173-bd6b-6a2744fbb067-kube-api-access-pbdh4\") pod \"payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh\" (UID: \"84563778-dd9f-4173-bd6b-6a2744fbb067\") " pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:14.638622 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.638392    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72c5x\" (UniqueName: \"kubernetes.io/projected/f2260124-5706-4fbe-865e-faf9ebe783a7-kube-api-access-72c5x\") pod \"payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5\" (UID: \"f2260124-5706-4fbe-865e-faf9ebe783a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:14.638622 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.638423    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2260124-5706-4fbe-865e-faf9ebe783a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5\" (UID: \"f2260124-5706-4fbe-865e-faf9ebe783a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:14.639034 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.639011    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84563778-dd9f-4173-bd6b-6a2744fbb067-plugins-config-volume\") pod \"payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh\" (UID: \"84563778-dd9f-4173-bd6b-6a2744fbb067\") " pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:14.639340 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.639318    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2260124-5706-4fbe-865e-faf9ebe783a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5\" (UID: \"f2260124-5706-4fbe-865e-faf9ebe783a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:14.647771 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.647744    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdh4\" (UniqueName: \"kubernetes.io/projected/84563778-dd9f-4173-bd6b-6a2744fbb067-kube-api-access-pbdh4\") pod \"payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh\" (UID: \"84563778-dd9f-4173-bd6b-6a2744fbb067\") " pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:14.648665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.648629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72c5x\" (UniqueName: \"kubernetes.io/projected/f2260124-5706-4fbe-865e-faf9ebe783a7-kube-api-access-72c5x\") pod \"payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5\" (UID: \"f2260124-5706-4fbe-865e-faf9ebe783a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:14.758041 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.758010    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:14.790365 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:14.790329    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:15.035094 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.035068    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"]
Sep 09 21:50:15.038131 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:50:15.038088    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2260124_5706_4fbe_865e_faf9ebe783a7.slice/crio-72f5448be9f4d04877b5fb05ffe7f27a8e4223367cc33ecc11b1d7f64a9b2dc7 WatchSource:0}: Error finding container 72f5448be9f4d04877b5fb05ffe7f27a8e4223367cc33ecc11b1d7f64a9b2dc7: Status 404 returned error can't find the container with id 72f5448be9f4d04877b5fb05ffe7f27a8e4223367cc33ecc11b1d7f64a9b2dc7
Sep 09 21:50:15.074236 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.074196    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"]
Sep 09 21:50:15.077613 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:50:15.077591    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84563778_dd9f_4173_bd6b_6a2744fbb067.slice/crio-409f810f34f8acf29ec5067c4d7cbec98bb0c106eb9670fc67269be02d239d7e WatchSource:0}: Error finding container 409f810f34f8acf29ec5067c4d7cbec98bb0c106eb9670fc67269be02d239d7e: Status 404 returned error can't find the container with id 409f810f34f8acf29ec5067c4d7cbec98bb0c106eb9670fc67269be02d239d7e
Sep 09 21:50:15.199802 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.199764    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:15.933927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.933879    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh" event={"ID":"84563778-dd9f-4173-bd6b-6a2744fbb067","Type":"ContainerStarted","Data":"3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d"}
Sep 09 21:50:15.934147 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.933950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh" event={"ID":"84563778-dd9f-4173-bd6b-6a2744fbb067","Type":"ContainerStarted","Data":"409f810f34f8acf29ec5067c4d7cbec98bb0c106eb9670fc67269be02d239d7e"}
Sep 09 21:50:15.934147 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.933980    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:15.935754 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.935724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5" event={"ID":"f2260124-5706-4fbe-865e-faf9ebe783a7","Type":"ContainerStarted","Data":"f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60"}
Sep 09 21:50:15.935892 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.935768    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5" event={"ID":"f2260124-5706-4fbe-865e-faf9ebe783a7","Type":"ContainerStarted","Data":"72f5448be9f4d04877b5fb05ffe7f27a8e4223367cc33ecc11b1d7f64a9b2dc7"}
Sep 09 21:50:15.935892 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.935812    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:15.954806 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.954746    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh" podStartSLOduration=1.954726844 podStartE2EDuration="1.954726844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:15.949365961 +0000 UTC m=+1018.323689786" watchObservedRunningTime="2026-09-09 21:50:15.954726844 +0000 UTC m=+1018.329050679"
Sep 09 21:50:15.976448 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:15.976320    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5" podStartSLOduration=1.9762988030000002 podStartE2EDuration="1.976298803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:15.974911286 +0000 UTC m=+1018.349235110" watchObservedRunningTime="2026-09-09 21:50:15.976298803 +0000 UTC m=+1018.350622628"
Sep 09 21:50:26.943900 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:26.943868    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:26.944368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:26.944063    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:37.637065 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:37.637022    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:38.276525 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:38.276491    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:38.371473 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:38.371438    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:38.665611 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:38.665584    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:44.483075 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:44.483042    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:44.581475 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:44.581443    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:55.237036 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.237005    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:55.257340 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.257305    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"]
Sep 09 21:50:55.257661 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.257630    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh" podUID="84563778-dd9f-4173-bd6b-6a2744fbb067" containerName="payload-processing" containerID="cri-o://3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d" gracePeriod=30
Sep 09 21:50:55.265914 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.265490    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"]
Sep 09 21:50:55.265914 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.265728    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5" podUID="f2260124-5706-4fbe-865e-faf9ebe783a7" containerName="payload-pre-processing" containerID="cri-o://f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60" gracePeriod=30
Sep 09 21:50:55.298605 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.298570    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:55.369248 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.369160    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:55.431962 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.431530    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:55.520673 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.520644    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:55.536825 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.536785    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84563778-dd9f-4173-bd6b-6a2744fbb067-plugins-config-volume\") pod \"84563778-dd9f-4173-bd6b-6a2744fbb067\" (UID: \"84563778-dd9f-4173-bd6b-6a2744fbb067\") "
Sep 09 21:50:55.537562 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.536858    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbdh4\" (UniqueName: \"kubernetes.io/projected/84563778-dd9f-4173-bd6b-6a2744fbb067-kube-api-access-pbdh4\") pod \"84563778-dd9f-4173-bd6b-6a2744fbb067\" (UID: \"84563778-dd9f-4173-bd6b-6a2744fbb067\") "
Sep 09 21:50:55.537814 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.537779    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84563778-dd9f-4173-bd6b-6a2744fbb067-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84563778-dd9f-4173-bd6b-6a2744fbb067" (UID: "84563778-dd9f-4173-bd6b-6a2744fbb067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:50:55.540317 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.540172    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84563778-dd9f-4173-bd6b-6a2744fbb067-kube-api-access-pbdh4" (OuterVolumeSpecName: "kube-api-access-pbdh4") pod "84563778-dd9f-4173-bd6b-6a2744fbb067" (UID: "84563778-dd9f-4173-bd6b-6a2744fbb067"). InnerVolumeSpecName "kube-api-access-pbdh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:50:55.550273 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.550253    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:55.638151 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.638121    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2260124-5706-4fbe-865e-faf9ebe783a7-plugins-config-volume\") pod \"f2260124-5706-4fbe-865e-faf9ebe783a7\" (UID: \"f2260124-5706-4fbe-865e-faf9ebe783a7\") "
Sep 09 21:50:55.638361 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.638283    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72c5x\" (UniqueName: \"kubernetes.io/projected/f2260124-5706-4fbe-865e-faf9ebe783a7-kube-api-access-72c5x\") pod \"f2260124-5706-4fbe-865e-faf9ebe783a7\" (UID: \"f2260124-5706-4fbe-865e-faf9ebe783a7\") "
Sep 09 21:50:55.638518 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.638495    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2260124-5706-4fbe-865e-faf9ebe783a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2260124-5706-4fbe-865e-faf9ebe783a7" (UID: "f2260124-5706-4fbe-865e-faf9ebe783a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:50:55.638596 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.638580    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84563778-dd9f-4173-bd6b-6a2744fbb067-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:50:55.638637 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.638603    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbdh4\" (UniqueName: \"kubernetes.io/projected/84563778-dd9f-4173-bd6b-6a2744fbb067-kube-api-access-pbdh4\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:50:55.640488 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.640466    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2260124-5706-4fbe-865e-faf9ebe783a7-kube-api-access-72c5x" (OuterVolumeSpecName: "kube-api-access-72c5x") pod "f2260124-5706-4fbe-865e-faf9ebe783a7" (UID: "f2260124-5706-4fbe-865e-faf9ebe783a7"). InnerVolumeSpecName "kube-api-access-72c5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:50:55.739728 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.739698    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2260124-5706-4fbe-865e-faf9ebe783a7-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:50:55.739728 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.739726    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72c5x\" (UniqueName: \"kubernetes.io/projected/f2260124-5706-4fbe-865e-faf9ebe783a7-kube-api-access-72c5x\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:50:55.758686 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:55.758584    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:56.154935 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.154906    2585 generic.go:356] "Generic (PLEG): container finished" podID="f2260124-5706-4fbe-865e-faf9ebe783a7" containerID="f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60" exitCode=0
Sep 09 21:50:56.155083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.154947    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"
Sep 09 21:50:56.155083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.154987    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5" event={"ID":"f2260124-5706-4fbe-865e-faf9ebe783a7","Type":"ContainerDied","Data":"f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60"}
Sep 09 21:50:56.155083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.155065    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5" event={"ID":"f2260124-5706-4fbe-865e-faf9ebe783a7","Type":"ContainerDied","Data":"72f5448be9f4d04877b5fb05ffe7f27a8e4223367cc33ecc11b1d7f64a9b2dc7"}
Sep 09 21:50:56.155290 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.155108    2585 scope.go:117] "RemoveContainer" containerID="f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60"
Sep 09 21:50:56.157745 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.156752    2585 generic.go:356] "Generic (PLEG): container finished" podID="84563778-dd9f-4173-bd6b-6a2744fbb067" containerID="3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d" exitCode=0
Sep 09 21:50:56.157745 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.156792    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh" event={"ID":"84563778-dd9f-4173-bd6b-6a2744fbb067","Type":"ContainerDied","Data":"3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d"}
Sep 09 21:50:56.157745 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.156833    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh" event={"ID":"84563778-dd9f-4173-bd6b-6a2744fbb067","Type":"ContainerDied","Data":"409f810f34f8acf29ec5067c4d7cbec98bb0c106eb9670fc67269be02d239d7e"}
Sep 09 21:50:56.157745 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.156899    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"
Sep 09 21:50:56.165662 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.165637    2585 scope.go:117] "RemoveContainer" containerID="f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60"
Sep 09 21:50:56.165982 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:50:56.165962    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60\": container with ID starting with f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60 not found: ID does not exist" containerID="f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60"
Sep 09 21:50:56.166045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.165992    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60"} err="failed to get container status \"f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60\": rpc error: code = NotFound desc = could not find container \"f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60\": container with ID starting with f2a9877f51d178cafa8fe0bca23fbeb22c3ac469b08399fc5aba75e7c8b05e60 not found: ID does not exist"
Sep 09 21:50:56.166045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.166010    2585 scope.go:117] "RemoveContainer" containerID="3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d"
Sep 09 21:50:56.174537 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.174520    2585 scope.go:117] "RemoveContainer" containerID="3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d"
Sep 09 21:50:56.174924 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:50:56.174894    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d\": container with ID starting with 3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d not found: ID does not exist" containerID="3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d"
Sep 09 21:50:56.175018 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.174933    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d"} err="failed to get container status \"3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d\": rpc error: code = NotFound desc = could not find container \"3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d\": container with ID starting with 3144c5efc6ded5d77d527d41e511fca6e7cc06da96390d2449d68b7358cd3f0d not found: ID does not exist"
Sep 09 21:50:56.175698 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.175672    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"]
Sep 09 21:50:56.177703 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.177678    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65d42a05-966449566-9mfd5"]
Sep 09 21:50:56.186520 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.186498    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"]
Sep 09 21:50:56.193263 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:56.193239    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65d42a05-5b84498bf7-9n2bh"]
Sep 09 21:50:58.128439 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:58.128403    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84563778-dd9f-4173-bd6b-6a2744fbb067" path="/var/lib/kubelet/pods/84563778-dd9f-4173-bd6b-6a2744fbb067/volumes"
Sep 09 21:50:58.128826 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:50:58.128783    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2260124-5706-4fbe-865e-faf9ebe783a7" path="/var/lib/kubelet/pods/f2260124-5706-4fbe-865e-faf9ebe783a7/volumes"
Sep 09 21:51:01.980299 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:01.980257    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:02.094495 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:02.094459    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:02.303351 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:02.303266    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:02.412031 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:02.411998    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:07.913491 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:07.913458    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:08.017605 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:08.017553    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:08.114073 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:08.114032    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:08.215281 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:08.215186    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:13.001450 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:13.001355    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:13.973152 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:13.973120    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:14.143624 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:14.143581    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:14.208194 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:14.208162    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:14.997160 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:14.997122    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:15.293822 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:15.293708    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:15.826994 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:15.826951    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:16.476571 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:16.476542    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:16.976447 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:16.976412    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:18.008257 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:18.008227    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:18.103434 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:18.103399    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:19.206959 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.206922    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:19.276724 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.276688    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:19.426592 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.426560    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:19.790527 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.790495    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:19.872695 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.872664    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:19.898468 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.898434    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"]
Sep 09 21:51:19.898853 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.898842    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2260124-5706-4fbe-865e-faf9ebe783a7" containerName="payload-pre-processing"
Sep 09 21:51:19.898901 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.898855    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2260124-5706-4fbe-865e-faf9ebe783a7" containerName="payload-pre-processing"
Sep 09 21:51:19.898901 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.898866    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84563778-dd9f-4173-bd6b-6a2744fbb067" containerName="payload-processing"
Sep 09 21:51:19.898901 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.898872    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="84563778-dd9f-4173-bd6b-6a2744fbb067" containerName="payload-processing"
Sep 09 21:51:19.898994 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.898938    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2260124-5706-4fbe-865e-faf9ebe783a7" containerName="payload-pre-processing"
Sep 09 21:51:19.898994 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.898947    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="84563778-dd9f-4173-bd6b-6a2744fbb067" containerName="payload-processing"
Sep 09 21:51:19.903478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.903462    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:19.905951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.905926    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d99c83-dockercfg-qc65w\""
Sep 09 21:51:19.906074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.906018    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d99c83\""
Sep 09 21:51:19.914087 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.914062    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"]
Sep 09 21:51:19.945148 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.945118    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"]
Sep 09 21:51:19.948819 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.948789    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:19.959518 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:19.959493    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"]
Sep 09 21:51:20.079359 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.079249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4kvj\" (UniqueName: \"kubernetes.io/projected/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-kube-api-access-h4kvj\") pod \"payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2\" (UID: \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:20.079535 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.079420    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4xz\" (UniqueName: \"kubernetes.io/projected/489d5c38-2b28-4baa-aab6-711c20df083a-kube-api-access-ss4xz\") pod \"payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf\" (UID: \"489d5c38-2b28-4baa-aab6-711c20df083a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:20.079535 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.079455    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2\" (UID: \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:20.079639 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.079546    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d5c38-2b28-4baa-aab6-711c20df083a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf\" (UID: \"489d5c38-2b28-4baa-aab6-711c20df083a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:20.180194 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.180155    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4kvj\" (UniqueName: \"kubernetes.io/projected/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-kube-api-access-h4kvj\") pod \"payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2\" (UID: \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:20.180384 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.180281    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4xz\" (UniqueName: \"kubernetes.io/projected/489d5c38-2b28-4baa-aab6-711c20df083a-kube-api-access-ss4xz\") pod \"payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf\" (UID: \"489d5c38-2b28-4baa-aab6-711c20df083a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:20.180384 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.180308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2\" (UID: \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:20.180532 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.180510    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d5c38-2b28-4baa-aab6-711c20df083a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf\" (UID: \"489d5c38-2b28-4baa-aab6-711c20df083a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:20.181055 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.181029    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2\" (UID: \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:20.181550 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.181529    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d5c38-2b28-4baa-aab6-711c20df083a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf\" (UID: \"489d5c38-2b28-4baa-aab6-711c20df083a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:20.187918 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.187897    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4xz\" (UniqueName: \"kubernetes.io/projected/489d5c38-2b28-4baa-aab6-711c20df083a-kube-api-access-ss4xz\") pod \"payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf\" (UID: \"489d5c38-2b28-4baa-aab6-711c20df083a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:20.188121 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.188099    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4kvj\" (UniqueName: \"kubernetes.io/projected/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-kube-api-access-h4kvj\") pod \"payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2\" (UID: \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:20.214740 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.214704    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:20.259927 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.259894    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:20.492123 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.492097    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"]
Sep 09 21:51:20.494258 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:51:20.494233    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4035eeb_b7f2_4ee7_b2b0_e77673665b1a.slice/crio-b3211fd0cbb06eeffa4ed52f460c5c9df8545578a4669437902718bd4443ebcb WatchSource:0}: Error finding container b3211fd0cbb06eeffa4ed52f460c5c9df8545578a4669437902718bd4443ebcb: Status 404 returned error can't find the container with id b3211fd0cbb06eeffa4ed52f460c5c9df8545578a4669437902718bd4443ebcb
Sep 09 21:51:20.553915 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:20.553889    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"]
Sep 09 21:51:20.556688 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:51:20.556664    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489d5c38_2b28_4baa_aab6_711c20df083a.slice/crio-2ffa3cef1121cb47d107a22110ae32cebe8848bc759ba0e799a97ad03166b50e WatchSource:0}: Error finding container 2ffa3cef1121cb47d107a22110ae32cebe8848bc759ba0e799a97ad03166b50e: Status 404 returned error can't find the container with id 2ffa3cef1121cb47d107a22110ae32cebe8848bc759ba0e799a97ad03166b50e
Sep 09 21:51:21.298486 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:21.298443    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2" event={"ID":"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a","Type":"ContainerStarted","Data":"708f19d7574d70a84097960639e2e41a06ef9b357db0b4ee424bb0c6d409b993"}
Sep 09 21:51:21.298976 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:21.298523    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:21.298976 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:21.298547    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2" event={"ID":"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a","Type":"ContainerStarted","Data":"b3211fd0cbb06eeffa4ed52f460c5c9df8545578a4669437902718bd4443ebcb"}
Sep 09 21:51:21.300726 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:21.300683    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf" event={"ID":"489d5c38-2b28-4baa-aab6-711c20df083a","Type":"ContainerStarted","Data":"8ebfec0dc02963df4698a16ba3655f04499e55e057397bfd51f1948c3529691c"}
Sep 09 21:51:21.300869 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:21.300756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf" event={"ID":"489d5c38-2b28-4baa-aab6-711c20df083a","Type":"ContainerStarted","Data":"2ffa3cef1121cb47d107a22110ae32cebe8848bc759ba0e799a97ad03166b50e"}
Sep 09 21:51:21.300869 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:21.300785    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:21.316427 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:21.316385    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2" podStartSLOduration=2.316370213 podStartE2EDuration="2.316370213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:21.313631681 +0000 UTC m=+1083.687955504" watchObservedRunningTime="2026-09-09 21:51:21.316370213 +0000 UTC m=+1083.690694072"
Sep 09 21:51:21.329532 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:21.329488    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf" podStartSLOduration=2.329473854 podStartE2EDuration="2.329473854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:21.328619057 +0000 UTC m=+1083.702942883" watchObservedRunningTime="2026-09-09 21:51:21.329473854 +0000 UTC m=+1083.703797677"
Sep 09 21:51:32.308799 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:32.308766    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:51:32.309227 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:32.308847    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:51:35.239074 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:35.239037    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:40.155480 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:40.155445    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:42.929664 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:42.929627    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:43.195055 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:43.194957    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:43.477199 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:43.477114    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:43.697710 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:43.697671    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:45.314997 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:45.314952    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:45.967844 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:45.967814    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:46.330321 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.330237    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:46.425895 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.425860    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:46.577276 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.577241    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"]
Sep 09 21:51:46.577560 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.577535    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk" podUID="2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" containerName="maas-api" containerID="cri-o://50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a" gracePeriod=30
Sep 09 21:51:46.599951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.599923    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:46.626731 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.626692    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"]
Sep 09 21:51:46.626967 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.626938    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5" podUID="b2f72e6a-21c8-4f97-b399-c798fb24f7f7" containerName="payload-processing" containerID="cri-o://ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f" gracePeriod=30
Sep 09 21:51:46.632373 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.632344    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"]
Sep 09 21:51:46.632584 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.632559    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2" podUID="7421895d-bbd4-4d48-a22d-7258bb6beb2f" containerName="payload-pre-processing" containerID="cri-o://2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb" gracePeriod=30
Sep 09 21:51:46.662099 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.662074    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:46.663138 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.663092    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2" podUID="7421895d-bbd4-4d48-a22d-7258bb6beb2f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.49:9004: connect: connection refused"
Sep 09 21:51:46.663138 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.663102    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk" podUID="2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused"
Sep 09 21:51:46.663328 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.663103    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5" podUID="b2f72e6a-21c8-4f97-b399-c798fb24f7f7" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.48:9004: connect: connection refused"
Sep 09 21:51:46.795463 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.795376    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:46.902631 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.902601    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:51:46.925438 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.925094    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:46.944367 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.941816    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:51:46.957171 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:46.957138    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:51:47.036149 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.036100    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:47.048253 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048230    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-plugins-config-volume\") pod \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\" (UID: \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\") "
Sep 09 21:51:47.048383 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048274    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ztm7\" (UniqueName: \"kubernetes.io/projected/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-kube-api-access-2ztm7\") pod \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") "
Sep 09 21:51:47.048383 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048316    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-metrics-certs\") pod \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") "
Sep 09 21:51:47.048383 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048378    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnffz\" (UniqueName: \"kubernetes.io/projected/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-kube-api-access-dnffz\") pod \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\" (UID: \"b2f72e6a-21c8-4f97-b399-c798fb24f7f7\") "
Sep 09 21:51:47.048498 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048415    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7421895d-bbd4-4d48-a22d-7258bb6beb2f-plugins-config-volume\") pod \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\" (UID: \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\") "
Sep 09 21:51:47.048498 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048459    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-openshift-service-ca\") pod \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") "
Sep 09 21:51:47.048498 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048487    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8khmc\" (UniqueName: \"kubernetes.io/projected/7421895d-bbd4-4d48-a22d-7258bb6beb2f-kube-api-access-8khmc\") pod \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\" (UID: \"7421895d-bbd4-4d48-a22d-7258bb6beb2f\") "
Sep 09 21:51:47.048637 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048549    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls\") pod \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\" (UID: \"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f\") "
Sep 09 21:51:47.048781 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048756    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2f72e6a-21c8-4f97-b399-c798fb24f7f7" (UID: "b2f72e6a-21c8-4f97-b399-c798fb24f7f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:51:47.049024 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.048869    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7421895d-bbd4-4d48-a22d-7258bb6beb2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7421895d-bbd4-4d48-a22d-7258bb6beb2f" (UID: "7421895d-bbd4-4d48-a22d-7258bb6beb2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:51:47.049120 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.049029    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" (UID: "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:51:47.050859 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.050823    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-kube-api-access-dnffz" (OuterVolumeSpecName: "kube-api-access-dnffz") pod "b2f72e6a-21c8-4f97-b399-c798fb24f7f7" (UID: "b2f72e6a-21c8-4f97-b399-c798fb24f7f7"). InnerVolumeSpecName "kube-api-access-dnffz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:51:47.051031 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.050994    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7421895d-bbd4-4d48-a22d-7258bb6beb2f-kube-api-access-8khmc" (OuterVolumeSpecName: "kube-api-access-8khmc") pod "7421895d-bbd4-4d48-a22d-7258bb6beb2f" (UID: "7421895d-bbd4-4d48-a22d-7258bb6beb2f"). InnerVolumeSpecName "kube-api-access-8khmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:51:47.051151 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.051128    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-kube-api-access-2ztm7" (OuterVolumeSpecName: "kube-api-access-2ztm7") pod "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" (UID: "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f"). InnerVolumeSpecName "kube-api-access-2ztm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:51:47.051151 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.051135    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" (UID: "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:51:47.051284 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.051262    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" (UID: "2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:51:47.149368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.149287    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:51:47.149368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.149313    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnffz\" (UniqueName: \"kubernetes.io/projected/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-kube-api-access-dnffz\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:51:47.149368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.149323    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7421895d-bbd4-4d48-a22d-7258bb6beb2f-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:51:47.149368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.149331    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:51:47.149368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.149340    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8khmc\" (UniqueName: \"kubernetes.io/projected/7421895d-bbd4-4d48-a22d-7258bb6beb2f-kube-api-access-8khmc\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:51:47.149368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.149348    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:51:47.149368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.149356    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f72e6a-21c8-4f97-b399-c798fb24f7f7-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:51:47.149368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.149363    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztm7\" (UniqueName: \"kubernetes.io/projected/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f-kube-api-access-2ztm7\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:51:47.241616 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.241581    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:47.448560 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.448517    2585 generic.go:356] "Generic (PLEG): container finished" podID="7421895d-bbd4-4d48-a22d-7258bb6beb2f" containerID="2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb" exitCode=0
Sep 09 21:51:47.449003 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.448569    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"
Sep 09 21:51:47.449003 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.448611    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2" event={"ID":"7421895d-bbd4-4d48-a22d-7258bb6beb2f","Type":"ContainerDied","Data":"2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb"}
Sep 09 21:51:47.449003 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.448676    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2" event={"ID":"7421895d-bbd4-4d48-a22d-7258bb6beb2f","Type":"ContainerDied","Data":"08207e97a0f089d3ef53b4e3cdf997e48e639def59fa707ebd5ef866cfbf6f25"}
Sep 09 21:51:47.449003 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.448706    2585 scope.go:117] "RemoveContainer" containerID="2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb"
Sep 09 21:51:47.450176 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.450156    2585 generic.go:356] "Generic (PLEG): container finished" podID="b2f72e6a-21c8-4f97-b399-c798fb24f7f7" containerID="ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f" exitCode=0
Sep 09 21:51:47.450300 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.450257    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"
Sep 09 21:51:47.450300 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.450262    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5" event={"ID":"b2f72e6a-21c8-4f97-b399-c798fb24f7f7","Type":"ContainerDied","Data":"ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f"}
Sep 09 21:51:47.450416 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.450320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5" event={"ID":"b2f72e6a-21c8-4f97-b399-c798fb24f7f7","Type":"ContainerDied","Data":"1ca676a2ad1dac6060a77c1565ec1880a21ecbdaa68d3fdcf40c48a6cb70dea6"}
Sep 09 21:51:47.452195 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.452170    2585 generic.go:356] "Generic (PLEG): container finished" podID="2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" containerID="50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a" exitCode=0
Sep 09 21:51:47.452349 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.452245    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"
Sep 09 21:51:47.452450 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.452430    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk" event={"ID":"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f","Type":"ContainerDied","Data":"50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a"}
Sep 09 21:51:47.452564 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.452545    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk" event={"ID":"2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f","Type":"ContainerDied","Data":"b84481e82b986c9b6b81bc5f2b7eed70e80c9590c8064bf402d50052f490f606"}
Sep 09 21:51:47.460739 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.460711    2585 scope.go:117] "RemoveContainer" containerID="2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb"
Sep 09 21:51:47.461006 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:51:47.460983    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb\": container with ID starting with 2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb not found: ID does not exist" containerID="2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb"
Sep 09 21:51:47.461096 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.461019    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb"} err="failed to get container status \"2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb\": rpc error: code = NotFound desc = could not find container \"2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb\": container with ID starting with 2ff075025e7880cf32a9667d065aacad987383a7d1593068c4e0fcdcefc747fb not found: ID does not exist"
Sep 09 21:51:47.461096 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.461042    2585 scope.go:117] "RemoveContainer" containerID="ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f"
Sep 09 21:51:47.469562 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.469544    2585 scope.go:117] "RemoveContainer" containerID="ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f"
Sep 09 21:51:47.469844 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:51:47.469826    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f\": container with ID starting with ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f not found: ID does not exist" containerID="ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f"
Sep 09 21:51:47.469915 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.469852    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f"} err="failed to get container status \"ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f\": rpc error: code = NotFound desc = could not find container \"ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f\": container with ID starting with ac85a6962cf0a1bfae0e6d9ad3186e5fc45c3dc7e7f6005957e3363bc585780f not found: ID does not exist"
Sep 09 21:51:47.469915 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.469867    2585 scope.go:117] "RemoveContainer" containerID="50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a"
Sep 09 21:51:47.474665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.474575    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"]
Sep 09 21:51:47.477907 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.477883    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-331c9d-64f4495494-qxll2"]
Sep 09 21:51:47.479907 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.479892    2585 scope.go:117] "RemoveContainer" containerID="50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a"
Sep 09 21:51:47.480187 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:51:47.480165    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a\": container with ID starting with 50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a not found: ID does not exist" containerID="50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a"
Sep 09 21:51:47.480314 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.480193    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a"} err="failed to get container status \"50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a\": rpc error: code = NotFound desc = could not find container \"50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a\": container with ID starting with 50e53b514141627697a47c5a0830380b12e3ebc380c6a26cbdc472f7f5a6106a not found: ID does not exist"
Sep 09 21:51:47.487155 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.487135    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"]
Sep 09 21:51:47.491498 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.491478    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-331c9d-7f6dc47dd6-zlwfk"]
Sep 09 21:51:47.501120 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.501101    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"]
Sep 09 21:51:47.504835 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.504815    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-331c9d-b5d7767c-mmtf5"]
Sep 09 21:51:47.567466 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.567416    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"]
Sep 09 21:51:47.568011 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.567987    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7421895d-bbd4-4d48-a22d-7258bb6beb2f" containerName="payload-pre-processing"
Sep 09 21:51:47.568011 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.568006    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7421895d-bbd4-4d48-a22d-7258bb6beb2f" containerName="payload-pre-processing"
Sep 09 21:51:47.568264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.568029    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f72e6a-21c8-4f97-b399-c798fb24f7f7" containerName="payload-processing"
Sep 09 21:51:47.568264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.568037    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f72e6a-21c8-4f97-b399-c798fb24f7f7" containerName="payload-processing"
Sep 09 21:51:47.568264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.568059    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" containerName="maas-api"
Sep 09 21:51:47.568264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.568067    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" containerName="maas-api"
Sep 09 21:51:47.568264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.568141    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f72e6a-21c8-4f97-b399-c798fb24f7f7" containerName="payload-processing"
Sep 09 21:51:47.568264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.568156    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7421895d-bbd4-4d48-a22d-7258bb6beb2f" containerName="payload-pre-processing"
Sep 09 21:51:47.568264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.568164    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" containerName="maas-api"
Sep 09 21:51:47.573169 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.573147    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.576381 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.576361    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-9e275c-serving-cert\""
Sep 09 21:51:47.583656 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.583617    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"]
Sep 09 21:51:47.612146 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.612116    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:47.654955 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.654927    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4n7\" (UniqueName: \"kubernetes.io/projected/030f7245-2aa2-41c4-abf7-6cbe13aa0308-kube-api-access-9m4n7\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.655137 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.654968    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.655137 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.654987    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030f7245-2aa2-41c4-abf7-6cbe13aa0308-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.655137 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.655079    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.710079 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.710001    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:47.755982 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.755949    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4n7\" (UniqueName: \"kubernetes.io/projected/030f7245-2aa2-41c4-abf7-6cbe13aa0308-kube-api-access-9m4n7\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.756149 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.755992    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.756149 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.756012    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030f7245-2aa2-41c4-abf7-6cbe13aa0308-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.756149 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.756070    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.756149 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:51:47.756117    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-serving-cert: secret "maas-api-e2e-shared-b-w2-9e275c-serving-cert" not found
Sep 09 21:51:47.756318 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:51:47.756198    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls podName:030f7245-2aa2-41c4-abf7-6cbe13aa0308 nodeName:}" failed. No retries permitted until 2026-09-09 21:51:48.256177422 +0000 UTC m=+1110.630501240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls") pod "maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww" (UID: "030f7245-2aa2-41c4-abf7-6cbe13aa0308") : secret "maas-api-e2e-shared-b-w2-9e275c-serving-cert" not found
Sep 09 21:51:47.756871 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.756838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030f7245-2aa2-41c4-abf7-6cbe13aa0308-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.758686 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.758662    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:47.766672 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:47.766652    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4n7\" (UniqueName: \"kubernetes.io/projected/030f7245-2aa2-41c4-abf7-6cbe13aa0308-kube-api-access-9m4n7\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:48.001789 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.001709    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:48.105222 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.105171    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:48.129836 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.129807    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f" path="/var/lib/kubelet/pods/2713ab25-b8c3-4b8f-bae2-fd86f3cd3e1f/volumes"
Sep 09 21:51:48.130193 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.130178    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7421895d-bbd4-4d48-a22d-7258bb6beb2f" path="/var/lib/kubelet/pods/7421895d-bbd4-4d48-a22d-7258bb6beb2f/volumes"
Sep 09 21:51:48.130560 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.130543    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f72e6a-21c8-4f97-b399-c798fb24f7f7" path="/var/lib/kubelet/pods/b2f72e6a-21c8-4f97-b399-c798fb24f7f7/volumes"
Sep 09 21:51:48.202128 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.202097    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:48.233645 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.233610    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"]
Sep 09 21:51:48.238509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.238490    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:48.240975 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.240953    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-9e275c-dockercfg-m5bsf\""
Sep 09 21:51:48.241330 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.241313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9e275c\""
Sep 09 21:51:48.249635 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.249611    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"]
Sep 09 21:51:48.261944 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.261874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:48.264651 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.264559    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:48.278082 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.278059    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"]
Sep 09 21:51:48.281569 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.281554    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:48.293712 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.293685    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"]
Sep 09 21:51:48.363168 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.363125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j5b\" (UniqueName: \"kubernetes.io/projected/d0435875-19c9-48be-ad5e-43e7715f2794-kube-api-access-76j5b\") pod \"payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9\" (UID: \"d0435875-19c9-48be-ad5e-43e7715f2794\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:48.363401 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.363174    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16578d45-ca6a-4d5e-b801-4da7e08e0f16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2\" (UID: \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:48.363401 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.363334    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0435875-19c9-48be-ad5e-43e7715f2794-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9\" (UID: \"d0435875-19c9-48be-ad5e-43e7715f2794\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:48.363535 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.363463    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpvhk\" (UniqueName: \"kubernetes.io/projected/16578d45-ca6a-4d5e-b801-4da7e08e0f16-kube-api-access-qpvhk\") pod \"payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2\" (UID: \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:48.464839 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.464808    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j5b\" (UniqueName: \"kubernetes.io/projected/d0435875-19c9-48be-ad5e-43e7715f2794-kube-api-access-76j5b\") pod \"payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9\" (UID: \"d0435875-19c9-48be-ad5e-43e7715f2794\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:48.465350 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.464852    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16578d45-ca6a-4d5e-b801-4da7e08e0f16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2\" (UID: \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:48.465350 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.464890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0435875-19c9-48be-ad5e-43e7715f2794-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9\" (UID: \"d0435875-19c9-48be-ad5e-43e7715f2794\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:48.465350 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.464992    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpvhk\" (UniqueName: \"kubernetes.io/projected/16578d45-ca6a-4d5e-b801-4da7e08e0f16-kube-api-access-qpvhk\") pod \"payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2\" (UID: \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:48.465660 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.465641    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16578d45-ca6a-4d5e-b801-4da7e08e0f16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2\" (UID: \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:48.465729 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.465709    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0435875-19c9-48be-ad5e-43e7715f2794-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9\" (UID: \"d0435875-19c9-48be-ad5e-43e7715f2794\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:48.473684 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.473652    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpvhk\" (UniqueName: \"kubernetes.io/projected/16578d45-ca6a-4d5e-b801-4da7e08e0f16-kube-api-access-qpvhk\") pod \"payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2\" (UID: \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:48.473811 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.473650    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j5b\" (UniqueName: \"kubernetes.io/projected/d0435875-19c9-48be-ad5e-43e7715f2794-kube-api-access-76j5b\") pod \"payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9\" (UID: \"d0435875-19c9-48be-ad5e-43e7715f2794\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:48.513596 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.513509    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:48.550008 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.549966    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:48.593593 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.593559    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:48.795655 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.795632    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"]
Sep 09 21:51:48.798347 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:51:48.798223    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030f7245_2aa2_41c4_abf7_6cbe13aa0308.slice/crio-c45f228341bdfbd4ae33f0230097111ce4ac1fff4b4369c7b503954a872edf83 WatchSource:0}: Error finding container c45f228341bdfbd4ae33f0230097111ce4ac1fff4b4369c7b503954a872edf83: Status 404 returned error can't find the container with id c45f228341bdfbd4ae33f0230097111ce4ac1fff4b4369c7b503954a872edf83
Sep 09 21:51:48.839678 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.837343    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"]
Sep 09 21:51:48.847985 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:51:48.847945    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0435875_19c9_48be_ad5e_43e7715f2794.slice/crio-2eebb08657550d102d323b262b80707aa09e280ad7a95f6578bf1c817c61347c WatchSource:0}: Error finding container 2eebb08657550d102d323b262b80707aa09e280ad7a95f6578bf1c817c61347c: Status 404 returned error can't find the container with id 2eebb08657550d102d323b262b80707aa09e280ad7a95f6578bf1c817c61347c
Sep 09 21:51:48.907005 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:48.906979    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"]
Sep 09 21:51:48.909818 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:51:48.909793    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16578d45_ca6a_4d5e_b801_4da7e08e0f16.slice/crio-f234ff071409c117d9a70c9cad96586d33e72667a67799a50d8afab1b2030245 WatchSource:0}: Error finding container f234ff071409c117d9a70c9cad96586d33e72667a67799a50d8afab1b2030245: Status 404 returned error can't find the container with id f234ff071409c117d9a70c9cad96586d33e72667a67799a50d8afab1b2030245
Sep 09 21:51:49.469512 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.468557    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9" event={"ID":"d0435875-19c9-48be-ad5e-43e7715f2794","Type":"ContainerStarted","Data":"d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b"}
Sep 09 21:51:49.469512 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.468625    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9" event={"ID":"d0435875-19c9-48be-ad5e-43e7715f2794","Type":"ContainerStarted","Data":"2eebb08657550d102d323b262b80707aa09e280ad7a95f6578bf1c817c61347c"}
Sep 09 21:51:49.469512 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.469459    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:51:49.472121 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.472094    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww" event={"ID":"030f7245-2aa2-41c4-abf7-6cbe13aa0308","Type":"ContainerStarted","Data":"578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1"}
Sep 09 21:51:49.472301 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.472143    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww" event={"ID":"030f7245-2aa2-41c4-abf7-6cbe13aa0308","Type":"ContainerStarted","Data":"c45f228341bdfbd4ae33f0230097111ce4ac1fff4b4369c7b503954a872edf83"}
Sep 09 21:51:49.472301 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.472171    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:51:49.474110 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.474085    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2" event={"ID":"16578d45-ca6a-4d5e-b801-4da7e08e0f16","Type":"ContainerStarted","Data":"6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69"}
Sep 09 21:51:49.474240 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.474122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2" event={"ID":"16578d45-ca6a-4d5e-b801-4da7e08e0f16","Type":"ContainerStarted","Data":"f234ff071409c117d9a70c9cad96586d33e72667a67799a50d8afab1b2030245"}
Sep 09 21:51:49.474240 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.474224    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:51:49.486546 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.486467    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9" podStartSLOduration=1.486454272 podStartE2EDuration="1.486454272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:49.485305624 +0000 UTC m=+1111.859629451" watchObservedRunningTime="2026-09-09 21:51:49.486454272 +0000 UTC m=+1111.860778095"
Sep 09 21:51:49.501038 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.500990    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2" podStartSLOduration=1.500969436 podStartE2EDuration="1.500969436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:49.499429293 +0000 UTC m=+1111.873753118" watchObservedRunningTime="2026-09-09 21:51:49.500969436 +0000 UTC m=+1111.875293259"
Sep 09 21:51:49.515851 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:49.515798    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww" podStartSLOduration=2.081122255 podStartE2EDuration="2.51578508s" totalImagesPullingTime="434.662825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:49.514059954 +0000 UTC m=+1111.888383796" watchObservedRunningTime="2026-09-09 21:51:49.51578508 +0000 UTC m=+1111.890108904"
Sep 09 21:51:52.248037 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:52.247994    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:55.486283 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:51:55.486251    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 21:52:00.482955 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:00.482926    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 21:52:01.488388 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:01.488354    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 21:52:11.225797 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:11.225757    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:19.922410 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:19.922375    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:20.766569 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:20.766533    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:21.046936 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:21.046847    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:22.140083 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:22.140025    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:24.509311 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:24.509262    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:24.605357 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:24.605323    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:25.673155 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:25.673097    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:25.774356 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:25.774319    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:25.982876 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:25.982790    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:26.142761 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.142724    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:26.246256 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.246126    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:26.255534 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.255503    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"]
Sep 09 21:52:26.259400 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.259377    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:26.262007 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.261988    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4883373b\""
Sep 09 21:52:26.262233 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.262202    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4883373b-dockercfg-zf4jn\""
Sep 09 21:52:26.269834 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.269813    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"]
Sep 09 21:52:26.275398 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.275376    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"]
Sep 09 21:52:26.279008 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.278988    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:26.294177 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.294149    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"]
Sep 09 21:52:26.328636 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.328604    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx\" (UID: \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:26.328814 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.328657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld48n\" (UniqueName: \"kubernetes.io/projected/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-kube-api-access-ld48n\") pod \"payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx\" (UID: \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:26.429658 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.429613    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-plugins-config-volume\") pod \"payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp\" (UID: \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\") " pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:26.429822 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.429689    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx\" (UID: \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:26.429822 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.429758    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld48n\" (UniqueName: \"kubernetes.io/projected/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-kube-api-access-ld48n\") pod \"payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx\" (UID: \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:26.429914 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.429851    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86855\" (UniqueName: \"kubernetes.io/projected/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-kube-api-access-86855\") pod \"payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp\" (UID: \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\") " pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:26.430441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.430417    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx\" (UID: \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:26.437940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.437907    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld48n\" (UniqueName: \"kubernetes.io/projected/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-kube-api-access-ld48n\") pod \"payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx\" (UID: \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:26.530790 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.530685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-plugins-config-volume\") pod \"payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp\" (UID: \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\") " pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:26.530973 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.530860    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86855\" (UniqueName: \"kubernetes.io/projected/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-kube-api-access-86855\") pod \"payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp\" (UID: \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\") " pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:26.531538 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.531514    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-plugins-config-volume\") pod \"payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp\" (UID: \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\") " pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:26.539378 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.539353    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86855\" (UniqueName: \"kubernetes.io/projected/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-kube-api-access-86855\") pod \"payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp\" (UID: \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\") " pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:26.572092 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.572060    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:26.592341 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.592317    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:26.847392 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.847316    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"]
Sep 09 21:52:26.850902 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:52:26.850877    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd61bd4d_dfc9_471b_9189_e9cc83da5f4c.slice/crio-626bd29b5accab39ec7eea50bda76617dba20d9d0a99c711c9912b2efa700780 WatchSource:0}: Error finding container 626bd29b5accab39ec7eea50bda76617dba20d9d0a99c711c9912b2efa700780: Status 404 returned error can't find the container with id 626bd29b5accab39ec7eea50bda76617dba20d9d0a99c711c9912b2efa700780
Sep 09 21:52:26.880951 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:26.880931    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"]
Sep 09 21:52:26.883361 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:52:26.883335    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ee3bfa_b87f_4af8_9eb1_54b33c2c22af.slice/crio-a3a69b90c8c99a320cd0beee4d58baa5225be535c6913040f0e2422a697af753 WatchSource:0}: Error finding container a3a69b90c8c99a320cd0beee4d58baa5225be535c6913040f0e2422a697af753: Status 404 returned error can't find the container with id a3a69b90c8c99a320cd0beee4d58baa5225be535c6913040f0e2422a697af753
Sep 09 21:52:27.679969 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:27.679934    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp" event={"ID":"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af","Type":"ContainerStarted","Data":"e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21"}
Sep 09 21:52:27.680171 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:27.680030    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp" event={"ID":"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af","Type":"ContainerStarted","Data":"a3a69b90c8c99a320cd0beee4d58baa5225be535c6913040f0e2422a697af753"}
Sep 09 21:52:27.680171 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:27.680056    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:27.681718 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:27.681697    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx" event={"ID":"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c","Type":"ContainerStarted","Data":"f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804"}
Sep 09 21:52:27.681841 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:27.681733    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx" event={"ID":"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c","Type":"ContainerStarted","Data":"626bd29b5accab39ec7eea50bda76617dba20d9d0a99c711c9912b2efa700780"}
Sep 09 21:52:27.681841 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:27.681753    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:27.697695 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:27.697642    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp" podStartSLOduration=1.697623133 podStartE2EDuration="1.697623133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:27.696095299 +0000 UTC m=+1150.070419125" watchObservedRunningTime="2026-09-09 21:52:27.697623133 +0000 UTC m=+1150.071946962"
Sep 09 21:52:27.720599 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:27.720536    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx" podStartSLOduration=1.720516508 podStartE2EDuration="1.720516508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:27.719171932 +0000 UTC m=+1150.093495760" watchObservedRunningTime="2026-09-09 21:52:27.720516508 +0000 UTC m=+1150.094840336"
Sep 09 21:52:28.832776 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:28.832707    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:29.228522 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:29.228485    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:34.074042 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:34.074005    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:38.689389 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:38.689360    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:52:38.689819 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:38.689445    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:52:42.628811 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:42.628740    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:46.523999 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:46.523964    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:49.450417 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:49.450372    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:51.966743 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:51.966702    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:52.070701 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:52.070659    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:52.228863 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:52.228779    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:52.650100 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:52.650018    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:52.945926 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:52.945894    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:53.154509 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:53.154472    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:54.156154 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:54.156122    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:54.260061 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:54.260019    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:54.356059 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:54.356028    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:58.793432 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:52:58.793392    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:10.968357 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:10.968316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:11.032863 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:11.032826    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:17.561111 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:17.561080    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:18.087315 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:18.087277    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:53:18.088427 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:18.088405    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:53:21.638146 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:21.638115    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:27.877862 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:27.877826    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:27.929308 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:27.929258    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:28.310414 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:28.310374    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:38.552729 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:38.552694    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:38.619186 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:38.619145    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:38.690382 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:38.690349    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:38.757665 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:38.757635    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:39.043305 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:39.043270    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:44.859748 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:44.859712    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:44.960553 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:44.960511    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:45.108451 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:45.108416    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:45.728454 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:45.728421    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:55.856759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:55.856723    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"]
Sep 09 21:53:55.857714 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:55.857650    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2" podUID="7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" containerName="maas-api" containerID="cri-o://afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906" gracePeriod=30
Sep 09 21:53:55.886982 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:55.886950    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:55.955258 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:55.955199    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:56.016472 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.016434    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:56.088804 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.088747    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:56.130410 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.130381    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:53:56.160638 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.160259    2585 generic.go:356] "Generic (PLEG): container finished" podID="7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" containerID="afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906" exitCode=0
Sep 09 21:53:56.160638 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.160323    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2" event={"ID":"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd","Type":"ContainerDied","Data":"afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906"}
Sep 09 21:53:56.160638 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.160472    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2" event={"ID":"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd","Type":"ContainerDied","Data":"c40c87e492392c13e67bb206e452b71fd0ee65c1d9ffe47aff2e50f8a2d13ef4"}
Sep 09 21:53:56.160638 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.160541    2585 scope.go:117] "RemoveContainer" containerID="afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906"
Sep 09 21:53:56.160638 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.160554    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"
Sep 09 21:53:56.161503 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.161475    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:56.171938 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.171917    2585 scope.go:117] "RemoveContainer" containerID="afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906"
Sep 09 21:53:56.172306 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:53:56.172285    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906\": container with ID starting with afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906 not found: ID does not exist" containerID="afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906"
Sep 09 21:53:56.172388 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.172317    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906"} err="failed to get container status \"afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906\": rpc error: code = NotFound desc = could not find container \"afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906\": container with ID starting with afcf14dabb58ecba32c35e1761b87c2dc9956ce8c410a0865cdea8e648237906 not found: ID does not exist"
Sep 09 21:53:56.216888 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.216843    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-metrics-certs\") pod \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") "
Sep 09 21:53:56.217089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.216901    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-openshift-service-ca\") pod \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") "
Sep 09 21:53:56.217089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.216997    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzzqm\" (UniqueName: \"kubernetes.io/projected/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-kube-api-access-xzzqm\") pod \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") "
Sep 09 21:53:56.217089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.217027    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-maas-api-tls\") pod \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\" (UID: \"7de6c8e4-0d4c-44e4-9a97-456fdce81fbd\") "
Sep 09 21:53:56.217973 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.217936    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" (UID: "7de6c8e4-0d4c-44e4-9a97-456fdce81fbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:53:56.219582 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.219555    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" (UID: "7de6c8e4-0d4c-44e4-9a97-456fdce81fbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:53:56.219953 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.219926    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-kube-api-access-xzzqm" (OuterVolumeSpecName: "kube-api-access-xzzqm") pod "7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" (UID: "7de6c8e4-0d4c-44e4-9a97-456fdce81fbd"). InnerVolumeSpecName "kube-api-access-xzzqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:53:56.220175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.220151    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" (UID: "7de6c8e4-0d4c-44e4-9a97-456fdce81fbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:53:56.317864 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.317828    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzzqm\" (UniqueName: \"kubernetes.io/projected/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-kube-api-access-xzzqm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:53:56.317864 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.317860    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:53:56.317864 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.317870    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:53:56.318105 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.317878    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:53:56.431106 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.431076    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:56.482086 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.482054    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"]
Sep 09 21:53:56.485542 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.485519    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da0a18-5d9c65fcfb-vgkp2"]
Sep 09 21:53:56.577610 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:56.577570    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:58.128627 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:53:58.128594    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" path="/var/lib/kubelet/pods/7de6c8e4-0d4c-44e4-9a97-456fdce81fbd/volumes"
Sep 09 21:54:01.188031 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:01.187988    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:01.511476 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:01.511369    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:09.016355 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:09.016316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:25.568262 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:25.568182    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:27.508167 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:27.508116    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:32.383999 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:32.383954    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:32.599214 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:32.599183    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:32.719355 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:32.719316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:37.746920 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:37.746885    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:38.144264 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:38.144124    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:38.379262 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:38.379227    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:38.904623 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:38.904586    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:39.182461 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:39.182349    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:39.383668 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:39.383629    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:42.474563 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:42.474529    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:42.563641 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:42.563597    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:43.749937 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:43.749891    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:43.855798 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:43.855752    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:44.185823 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:44.185779    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:44.366300 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:44.366256    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:44.458781 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:44.458695    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:56.923414 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:54:56.923374    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:07.368889 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:07.368832    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:07.679950 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:07.679913    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:07.976531 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:07.976445    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:08.300312 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:08.300198    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:10.714401 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:10.714366    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:10.808627 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:10.808594    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:11.426600 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:11.426567    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:12.173187 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.173158    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"]
Sep 09 21:55:12.174417 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.174391    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" containerName="maas-api"
Sep 09 21:55:12.174417 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.174417    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" containerName="maas-api"
Sep 09 21:55:12.174571 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.174538    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de6c8e4-0d4c-44e4-9a97-456fdce81fbd" containerName="maas-api"
Sep 09 21:55:12.177990 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.177973    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.180653 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.180632    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-64fec6-serving-cert\""
Sep 09 21:55:12.187167 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.187131    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"]
Sep 09 21:55:12.216173 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.212385    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:12.269789 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.269746    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.270008 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.269843    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3087450-ec4c-47e5-bfcc-09b6424cb64e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.270008 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.269877    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.270008 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.269922    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stqt\" (UniqueName: \"kubernetes.io/projected/c3087450-ec4c-47e5-bfcc-09b6424cb64e-kube-api-access-4stqt\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.313366 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.313333    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:12.370853 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.370808    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.371247 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.371201    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3087450-ec4c-47e5-bfcc-09b6424cb64e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.371609 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.371588    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.372812 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.372778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stqt\" (UniqueName: \"kubernetes.io/projected/c3087450-ec4c-47e5-bfcc-09b6424cb64e-kube-api-access-4stqt\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.373017 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.372679    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3087450-ec4c-47e5-bfcc-09b6424cb64e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.374853 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.374829    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.375619 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.375589    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.383636 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.383597    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stqt\" (UniqueName: \"kubernetes.io/projected/c3087450-ec4c-47e5-bfcc-09b6424cb64e-kube-api-access-4stqt\") pod \"maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.412226 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.412183    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:12.493307 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.493190    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:12.778777 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.778752    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"]
Sep 09 21:55:12.781176 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:55:12.781148    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3087450_ec4c_47e5_bfcc_09b6424cb64e.slice/crio-3d409a2b7394a388b4b9745e5ef97492a55391e23659c8aa31bad25082a5ca30 WatchSource:0}: Error finding container 3d409a2b7394a388b4b9745e5ef97492a55391e23659c8aa31bad25082a5ca30: Status 404 returned error can't find the container with id 3d409a2b7394a388b4b9745e5ef97492a55391e23659c8aa31bad25082a5ca30
Sep 09 21:55:12.782402 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.782380    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:55:12.791021 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.790993    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:12.863265 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:12.863172    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:13.208907 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:13.208869    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:13.573731 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:13.573632    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g" event={"ID":"c3087450-ec4c-47e5-bfcc-09b6424cb64e","Type":"ContainerStarted","Data":"ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54"}
Sep 09 21:55:13.573731 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:13.573692    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g" event={"ID":"c3087450-ec4c-47e5-bfcc-09b6424cb64e","Type":"ContainerStarted","Data":"3d409a2b7394a388b4b9745e5ef97492a55391e23659c8aa31bad25082a5ca30"}
Sep 09 21:55:13.573953 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:13.573819    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:13.590431 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:13.590374    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g" podStartSLOduration=1.0690325999999999 podStartE2EDuration="1.590357836s" totalImagesPullingTime="521.325236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:13.588505895 +0000 UTC m=+1315.962829718" watchObservedRunningTime="2026-09-09 21:55:13.590357836 +0000 UTC m=+1315.964681677"
Sep 09 21:55:13.909537 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:13.909497    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:14.935794 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:14.935711    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:19.584484 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:19.584456    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 21:55:21.655744 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:21.655714    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:27.272505 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.272464    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"]
Sep 09 21:55:27.272984 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.272690    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp" podUID="e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af" containerName="payload-processing" containerID="cri-o://e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21" gracePeriod=30
Sep 09 21:55:27.278846 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.278811    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"]
Sep 09 21:55:27.279299 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.279265    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx" podUID="dd61bd4d-dfc9-471b-9189-e9cc83da5f4c" containerName="payload-pre-processing" containerID="cri-o://f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804" gracePeriod=30
Sep 09 21:55:27.283267 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.283155    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:27.347103 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.347068    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:27.439953 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.439902    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:27.508331 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.508290    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:27.571201 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.571174    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:55:27.574395 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.574372    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:55:27.630062 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.630032    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-plugins-config-volume\") pod \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\" (UID: \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\") "
Sep 09 21:55:27.630270 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.630110    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld48n\" (UniqueName: \"kubernetes.io/projected/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-kube-api-access-ld48n\") pod \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\" (UID: \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\") "
Sep 09 21:55:27.630270 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.630157    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86855\" (UniqueName: \"kubernetes.io/projected/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-kube-api-access-86855\") pod \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\" (UID: \"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af\") "
Sep 09 21:55:27.630270 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.630258    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-plugins-config-volume\") pod \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\" (UID: \"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c\") "
Sep 09 21:55:27.630583 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.630470    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af" (UID: "e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:27.630999 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.630962    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd61bd4d-dfc9-471b-9189-e9cc83da5f4c" (UID: "dd61bd4d-dfc9-471b-9189-e9cc83da5f4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:27.631174 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.631101    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:55:27.632782 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.632752    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-kube-api-access-ld48n" (OuterVolumeSpecName: "kube-api-access-ld48n") pod "dd61bd4d-dfc9-471b-9189-e9cc83da5f4c" (UID: "dd61bd4d-dfc9-471b-9189-e9cc83da5f4c"). InnerVolumeSpecName "kube-api-access-ld48n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:27.632899 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.632795    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-kube-api-access-86855" (OuterVolumeSpecName: "kube-api-access-86855") pod "e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af" (UID: "e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af"). InnerVolumeSpecName "kube-api-access-86855". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:27.654015 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.653975    2585 generic.go:356] "Generic (PLEG): container finished" podID="e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af" containerID="e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21" exitCode=0
Sep 09 21:55:27.654165 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.654027    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"
Sep 09 21:55:27.654165 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.654038    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp" event={"ID":"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af","Type":"ContainerDied","Data":"e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21"}
Sep 09 21:55:27.654165 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.654094    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp" event={"ID":"e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af","Type":"ContainerDied","Data":"a3a69b90c8c99a320cd0beee4d58baa5225be535c6913040f0e2422a697af753"}
Sep 09 21:55:27.654165 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.654126    2585 scope.go:117] "RemoveContainer" containerID="e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21"
Sep 09 21:55:27.655694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.655671    2585 generic.go:356] "Generic (PLEG): container finished" podID="dd61bd4d-dfc9-471b-9189-e9cc83da5f4c" containerID="f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804" exitCode=0
Sep 09 21:55:27.655808 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.655713    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"
Sep 09 21:55:27.655808 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.655752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx" event={"ID":"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c","Type":"ContainerDied","Data":"f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804"}
Sep 09 21:55:27.655808 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.655803    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx" event={"ID":"dd61bd4d-dfc9-471b-9189-e9cc83da5f4c","Type":"ContainerDied","Data":"626bd29b5accab39ec7eea50bda76617dba20d9d0a99c711c9912b2efa700780"}
Sep 09 21:55:27.666399 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.666248    2585 scope.go:117] "RemoveContainer" containerID="e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21"
Sep 09 21:55:27.666675 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:55:27.666652    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21\": container with ID starting with e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21 not found: ID does not exist" containerID="e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21"
Sep 09 21:55:27.666764 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.666687    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21"} err="failed to get container status \"e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21\": rpc error: code = NotFound desc = could not find container \"e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21\": container with ID starting with e88c9af399b6da3a547b48e7e169e735743582976d68bcab01cf7622999d4b21 not found: ID does not exist"
Sep 09 21:55:27.666764 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.666711    2585 scope.go:117] "RemoveContainer" containerID="f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804"
Sep 09 21:55:27.675132 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.675050    2585 scope.go:117] "RemoveContainer" containerID="f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804"
Sep 09 21:55:27.675867 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:55:27.675839    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804\": container with ID starting with f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804 not found: ID does not exist" containerID="f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804"
Sep 09 21:55:27.675981 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.675878    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804"} err="failed to get container status \"f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804\": rpc error: code = NotFound desc = could not find container \"f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804\": container with ID starting with f7e05c1443faa3ba692ac989b235fdf3df49310f03829a4cee0a55cea7b7e804 not found: ID does not exist"
Sep 09 21:55:27.677978 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.677957    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"]
Sep 09 21:55:27.686565 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.686546    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4883373b-7cfd55c54f-rqfkp"]
Sep 09 21:55:27.698125 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.698098    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"]
Sep 09 21:55:27.699950 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.699933    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4883373b-77b9d687bb-lwlzx"]
Sep 09 21:55:27.731846 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.731813    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld48n\" (UniqueName: \"kubernetes.io/projected/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-kube-api-access-ld48n\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:55:27.731846 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.731840    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86855\" (UniqueName: \"kubernetes.io/projected/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af-kube-api-access-86855\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:55:27.731846 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:27.731851    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:55:28.128468 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:28.128437    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd61bd4d-dfc9-471b-9189-e9cc83da5f4c" path="/var/lib/kubelet/pods/dd61bd4d-dfc9-471b-9189-e9cc83da5f4c/volumes"
Sep 09 21:55:28.128837 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:28.128824    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af" path="/var/lib/kubelet/pods/e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af/volumes"
Sep 09 21:55:30.442940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:30.442904    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:33.186346 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:33.186314    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:35.472625 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:35.472592    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:35.615743 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:35.615703    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:49.387353 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.387267    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"]
Sep 09 21:55:49.387940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.387735    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd61bd4d-dfc9-471b-9189-e9cc83da5f4c" containerName="payload-pre-processing"
Sep 09 21:55:49.387940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.387747    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd61bd4d-dfc9-471b-9189-e9cc83da5f4c" containerName="payload-pre-processing"
Sep 09 21:55:49.387940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.387762    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af" containerName="payload-processing"
Sep 09 21:55:49.387940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.387767    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af" containerName="payload-processing"
Sep 09 21:55:49.387940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.387836    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd61bd4d-dfc9-471b-9189-e9cc83da5f4c" containerName="payload-pre-processing"
Sep 09 21:55:49.387940 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.387846    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ee3bfa-b87f-4af8-9eb1-54b33c2c22af" containerName="payload-processing"
Sep 09 21:55:49.391501 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.391477    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.393673 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.393652    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"unconfigured-w4-3fa986-kserve-self-signed-certs\""
Sep 09 21:55:49.399324 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.399040    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"]
Sep 09 21:55:49.419679 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.419587    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:49.453099 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.453064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzkj\" (UniqueName: \"kubernetes.io/projected/92ceeeb1-782c-4638-a172-fff61580c2d3-kube-api-access-2fzkj\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.453296 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.453106    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-model-cache\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.453296 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.453130    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-tmp-dir\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.453296 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.453165    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-home\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.453296 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.453193    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ceeeb1-782c-4638-a172-fff61580c2d3-tls-certs\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.453296 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.453239    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-dshm\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.553884 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.553847    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzkj\" (UniqueName: \"kubernetes.io/projected/92ceeeb1-782c-4638-a172-fff61580c2d3-kube-api-access-2fzkj\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.553884 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.553890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-model-cache\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.554124 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.553909    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-tmp-dir\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.554124 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.553926    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-home\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.554124 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.553947    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ceeeb1-782c-4638-a172-fff61580c2d3-tls-certs\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.554124 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.553979    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-dshm\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.554368 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.554345    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-model-cache\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.554428 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.554365    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-tmp-dir\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.554428 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.554401    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-home\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.556439 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.556416    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-dshm\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.556532 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.556516    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ceeeb1-782c-4638-a172-fff61580c2d3-tls-certs\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.561244 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.561220    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzkj\" (UniqueName: \"kubernetes.io/projected/92ceeeb1-782c-4638-a172-fff61580c2d3-kube-api-access-2fzkj\") pod \"unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") " pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.701881 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.701838    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:49.977631 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:49.977607    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"]
Sep 09 21:55:49.980030 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:55:49.980001    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92ceeeb1_782c_4638_a172_fff61580c2d3.slice/crio-d86b2d4fc70445658a69d05e55405098f98dc0ff64c3fec2c7ce4ce8f794d3f2 WatchSource:0}: Error finding container d86b2d4fc70445658a69d05e55405098f98dc0ff64c3fec2c7ce4ce8f794d3f2: Status 404 returned error can't find the container with id d86b2d4fc70445658a69d05e55405098f98dc0ff64c3fec2c7ce4ce8f794d3f2
Sep 09 21:55:50.785254 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:50.785201    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8" event={"ID":"92ceeeb1-782c-4638-a172-fff61580c2d3","Type":"ContainerStarted","Data":"e49248493aef68adc7424aa713be51318d472d6f6c1a84e60bbb7b9559070ac1"}
Sep 09 21:55:50.785640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:50.785263    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8" event={"ID":"92ceeeb1-782c-4638-a172-fff61580c2d3","Type":"ContainerStarted","Data":"d86b2d4fc70445658a69d05e55405098f98dc0ff64c3fec2c7ce4ce8f794d3f2"}
Sep 09 21:55:50.800986 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:50.800936    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8" podStartSLOduration=1.800919873 podStartE2EDuration="1.800919873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:50.799750776 +0000 UTC m=+1353.174074604" watchObservedRunningTime="2026-09-09 21:55:50.800919873 +0000 UTC m=+1353.175243754"
Sep 09 21:55:59.702880 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:59.702841    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:59.703404 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:59.702918    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:59.716068 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:59.716040    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:55:59.845367 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:55:59.845329    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 21:56:06.029840 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:06.029806    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:11.700575 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:11.700542    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:12.456023 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:12.455982    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:13.775760 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:13.775704    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:14.844637 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:14.844604    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:18.136426 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:18.136393    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:30.583914 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:30.583885    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:40.834516 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:40.834483    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:51.087857 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:51.087819    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:51.352118 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:51.352018    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:51.666312 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:51.666281    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.862127 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:52.862077    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.977403 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:52.977368    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:53.669159 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:53.669129    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:53.770541 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:53.770508    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:53.985017 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:53.984923    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:55.131103 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:55.131070    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:58.149473 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:56:58.149444    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:01.252019 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:01.251963    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:01.352349 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:01.352317    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:01.925238 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:01.925188    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:07.812845 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:07.812813    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:07.915452 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:07.915418    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:08.012610 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:08.012573    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:08.151752 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:08.151668    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:09.002292 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:09.002260    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:12.752644 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:12.752604    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:12.817753 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:12.817722    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:18.553803 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:18.553771    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:18.767317 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:18.767284    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:27.696333 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.696301    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"]
Sep 09 21:57:27.699473 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.699456    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.703093 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.703070    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"kube-root-ca.crt\""
Sep 09 21:57:27.703242 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.703130    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"auth-test-model-64fec6-kserve-self-signed-certs\""
Sep 09 21:57:27.703242 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.703130    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"openshift-service-ca.crt\""
Sep 09 21:57:27.703438 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.703420    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"default-dockercfg-m9z8s\""
Sep 09 21:57:27.715769 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.715745    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"]
Sep 09 21:57:27.753694 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.753661    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:27.830222 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.830179    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqr55\" (UniqueName: \"kubernetes.io/projected/edadd607-4df0-44ea-982b-4209c2a1be30-kube-api-access-rqr55\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.830433 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.830295    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-tmp-dir\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.830433 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.830387    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-dshm\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.830433 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.830421    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edadd607-4df0-44ea-982b-4209c2a1be30-tls-certs\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.830605 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.830445    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-home\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.830605 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.830526    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-model-cache\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.931939 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.931908    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-model-cache\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.932158 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.931960    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqr55\" (UniqueName: \"kubernetes.io/projected/edadd607-4df0-44ea-982b-4209c2a1be30-kube-api-access-rqr55\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.932158 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.932091    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-tmp-dir\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.932158 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.932157    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-dshm\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.932408 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.932184    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edadd607-4df0-44ea-982b-4209c2a1be30-tls-certs\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.932408 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.932224    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-home\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.932494 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.932406    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-model-cache\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.932494 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.932429    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-tmp-dir\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.932601 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.932579    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-home\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.934539 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.934515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-dshm\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.934778 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.934762    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edadd607-4df0-44ea-982b-4209c2a1be30-tls-certs\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:27.941294 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:27.941269    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqr55\" (UniqueName: \"kubernetes.io/projected/edadd607-4df0-44ea-982b-4209c2a1be30-kube-api-access-rqr55\") pod \"auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:28.009360 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:28.009271    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:28.286097 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:28.286069    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"]
Sep 09 21:57:28.288163 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:57:28.288117    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedadd607_4df0_44ea_982b_4209c2a1be30.slice/crio-c141482b21cdd2a6c0c4c5b6c59a1fa1e3b49114713df9ea6680c49a1cbc429f WatchSource:0}: Error finding container c141482b21cdd2a6c0c4c5b6c59a1fa1e3b49114713df9ea6680c49a1cbc429f: Status 404 returned error can't find the container with id c141482b21cdd2a6c0c4c5b6c59a1fa1e3b49114713df9ea6680c49a1cbc429f
Sep 09 21:57:28.297912 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:28.297887    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc" event={"ID":"edadd607-4df0-44ea-982b-4209c2a1be30","Type":"ContainerStarted","Data":"c141482b21cdd2a6c0c4c5b6c59a1fa1e3b49114713df9ea6680c49a1cbc429f"}
Sep 09 21:57:29.312049 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:29.312001    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc" event={"ID":"edadd607-4df0-44ea-982b-4209c2a1be30","Type":"ContainerStarted","Data":"b5da73fb293bb3564828bd5928f278555c9e6348b16fd24a2c2967bfe9c3b998"}
Sep 09 21:57:29.331719 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:29.331662    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc" podStartSLOduration=2.331646498 podStartE2EDuration="2.331646498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:57:29.329832135 +0000 UTC m=+1451.704155960" watchObservedRunningTime="2026-09-09 21:57:29.331646498 +0000 UTC m=+1451.705970321"
Sep 09 21:57:31.012981 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:31.012944    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:31.115128 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:31.115090    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:34.815541 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:34.815483    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:38.009730 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:38.009694    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:38.010133 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:38.009750    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:38.022865 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:38.022834    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:38.369403 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:38.369319    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 21:57:39.772246 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:39.772192    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:46.147292 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:46.147255    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:46.233823 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:46.233787    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:46.493390 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:46.493334    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:46.787489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:46.787399    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:46.963131 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:46.963100    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:49.779474 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:49.779437    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:49.918741 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:49.918699    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:50.998090 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:50.998053    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:51.096814 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:51.096786    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:51.798279 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:51.798240    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:51.977419 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:51.977373    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:52.059732 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.059660    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"]
Sep 09 21:57:52.063104 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.063088    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:52.066323 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.066291    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5d8401-dockercfg-dcnbx\""
Sep 09 21:57:52.066755 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.066732    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5d8401\""
Sep 09 21:57:52.091622 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.091595    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"]
Sep 09 21:57:52.100745 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.100720    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"]
Sep 09 21:57:52.104126 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.104107    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:52.117905 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.117879    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"]
Sep 09 21:57:52.176076 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.176043    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d2fd\" (UniqueName: \"kubernetes.io/projected/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-kube-api-access-4d2fd\") pod \"payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg\" (UID: \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:52.176276 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.176083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78qh\" (UniqueName: \"kubernetes.io/projected/c0014150-3259-4647-b770-4e944b147ba3-kube-api-access-s78qh\") pod \"payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr\" (UID: \"c0014150-3259-4647-b770-4e944b147ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:52.176276 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.176119    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0014150-3259-4647-b770-4e944b147ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr\" (UID: \"c0014150-3259-4647-b770-4e944b147ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:52.176276 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.176256    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg\" (UID: \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:52.276826 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.276785    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d2fd\" (UniqueName: \"kubernetes.io/projected/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-kube-api-access-4d2fd\") pod \"payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg\" (UID: \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:52.277023 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.276835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78qh\" (UniqueName: \"kubernetes.io/projected/c0014150-3259-4647-b770-4e944b147ba3-kube-api-access-s78qh\") pod \"payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr\" (UID: \"c0014150-3259-4647-b770-4e944b147ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:52.277023 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.276867    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0014150-3259-4647-b770-4e944b147ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr\" (UID: \"c0014150-3259-4647-b770-4e944b147ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:52.277023 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.276939    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg\" (UID: \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:52.277759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.277731    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg\" (UID: \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:52.277892 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.277742    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0014150-3259-4647-b770-4e944b147ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr\" (UID: \"c0014150-3259-4647-b770-4e944b147ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:52.289088 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.289055    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d2fd\" (UniqueName: \"kubernetes.io/projected/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-kube-api-access-4d2fd\") pod \"payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg\" (UID: \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:52.289598 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.289573    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78qh\" (UniqueName: \"kubernetes.io/projected/c0014150-3259-4647-b770-4e944b147ba3-kube-api-access-s78qh\") pod \"payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr\" (UID: \"c0014150-3259-4647-b770-4e944b147ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:52.375246 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.375131    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:52.415407 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.415362    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:52.657195 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.657166    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"]
Sep 09 21:57:52.659247 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:57:52.659199    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3a06ef_f6bb_419c_9a38_bb1165c10dbb.slice/crio-da844f2af2e1c9866782f52f220de579566991f740bda25befdf4d1970d99ee5 WatchSource:0}: Error finding container da844f2af2e1c9866782f52f220de579566991f740bda25befdf4d1970d99ee5: Status 404 returned error can't find the container with id da844f2af2e1c9866782f52f220de579566991f740bda25befdf4d1970d99ee5
Sep 09 21:57:52.713441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:52.713406    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"]
Sep 09 21:57:52.716440 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:57:52.716385    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0014150_3259_4647_b770_4e944b147ba3.slice/crio-b5981e81700b73c1dd593ff40ca87e09887244268f18880cf650e4322ac07635 WatchSource:0}: Error finding container b5981e81700b73c1dd593ff40ca87e09887244268f18880cf650e4322ac07635: Status 404 returned error can't find the container with id b5981e81700b73c1dd593ff40ca87e09887244268f18880cf650e4322ac07635
Sep 09 21:57:53.442396 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:53.442360    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg" event={"ID":"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb","Type":"ContainerStarted","Data":"112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b"}
Sep 09 21:57:53.442864 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:53.442427    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:57:53.442864 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:53.442454    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg" event={"ID":"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb","Type":"ContainerStarted","Data":"da844f2af2e1c9866782f52f220de579566991f740bda25befdf4d1970d99ee5"}
Sep 09 21:57:53.444124 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:53.444101    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr" event={"ID":"c0014150-3259-4647-b770-4e944b147ba3","Type":"ContainerStarted","Data":"6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab"}
Sep 09 21:57:53.444282 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:53.444137    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr" event={"ID":"c0014150-3259-4647-b770-4e944b147ba3","Type":"ContainerStarted","Data":"b5981e81700b73c1dd593ff40ca87e09887244268f18880cf650e4322ac07635"}
Sep 09 21:57:53.444282 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:53.444252    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:57:53.470285 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:53.470234    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg" podStartSLOduration=1.4701986599999999 podStartE2EDuration="1.47019866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:57:53.468013859 +0000 UTC m=+1475.842337686" watchObservedRunningTime="2026-09-09 21:57:53.47019866 +0000 UTC m=+1475.844522505"
Sep 09 21:57:53.492438 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:53.492394    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr" podStartSLOduration=1.492378108 podStartE2EDuration="1.492378108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:57:53.489555536 +0000 UTC m=+1475.863879360" watchObservedRunningTime="2026-09-09 21:57:53.492378108 +0000 UTC m=+1475.866702001"
Sep 09 21:57:59.479753 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:59.479717    2585 generic.go:356] "Generic (PLEG): container finished" podID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerID="50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5" exitCode=137
Sep 09 21:57:59.480200 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:59.479795    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k" event={"ID":"90e25932-0d31-4c53-944a-4a3dd2093b20","Type":"ContainerDied","Data":"50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5"}
Sep 09 21:57:59.480200 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:57:59.480150    2585 scope.go:117] "RemoveContainer" containerID="50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5"
Sep 09 21:58:00.487843 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:00.487810    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k" event={"ID":"90e25932-0d31-4c53-944a-4a3dd2093b20","Type":"ContainerStarted","Data":"125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f"}
Sep 09 21:58:00.488257 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:00.488012    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:58:04.453333 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:04.453302    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:58:04.453736 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:04.453370    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:58:11.495278 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:11.495250    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 21:58:14.708289 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:14.708251    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:15.134233 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:15.134119    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:15.529295 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:15.529083    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:15.662482 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:15.662445    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:18.125923 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:18.125892    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:58:18.126416 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:18.125920    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 21:58:19.167478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:19.167448    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:19.286196 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:19.286153    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:20.345247 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:20.345185    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:20.434171 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:20.434134    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:21.017952 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.017918    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:21.186638 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.186603    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:21.228842 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.228802    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"]
Sep 09 21:58:21.232578 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.232555    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:21.239114 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.239094    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9dedad-dockercfg-jtsf4\""
Sep 09 21:58:21.239278 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.239166    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9dedad\""
Sep 09 21:58:21.254355 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.254330    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"]
Sep 09 21:58:21.263673 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.263642    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgjrj\" (UniqueName: \"kubernetes.io/projected/1a13fc89-e7b5-4bcb-b81f-d13f17498467-kube-api-access-jgjrj\") pod \"payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65\" (UID: \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:21.263798 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.263694    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a13fc89-e7b5-4bcb-b81f-d13f17498467-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65\" (UID: \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:21.364729 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.364642    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjrj\" (UniqueName: \"kubernetes.io/projected/1a13fc89-e7b5-4bcb-b81f-d13f17498467-kube-api-access-jgjrj\") pod \"payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65\" (UID: \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:21.364729 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.364692    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a13fc89-e7b5-4bcb-b81f-d13f17498467-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65\" (UID: \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:21.365588 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.365568    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a13fc89-e7b5-4bcb-b81f-d13f17498467-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65\" (UID: \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:21.374872 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.374846    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgjrj\" (UniqueName: \"kubernetes.io/projected/1a13fc89-e7b5-4bcb-b81f-d13f17498467-kube-api-access-jgjrj\") pod \"payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65\" (UID: \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:21.543898 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.543864    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:21.822014 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:21.821991    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"]
Sep 09 21:58:21.824320 ip-10-0-128-94 kubenswrapper[2585]: W0909 21:58:21.824267    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a13fc89_e7b5_4bcb_b81f_d13f17498467.slice/crio-9c0364a5e59784b6fdcf5de6ecae019b851d86de2998a84997fa4316a3c1217d WatchSource:0}: Error finding container 9c0364a5e59784b6fdcf5de6ecae019b851d86de2998a84997fa4316a3c1217d: Status 404 returned error can't find the container with id 9c0364a5e59784b6fdcf5de6ecae019b851d86de2998a84997fa4316a3c1217d
Sep 09 21:58:22.613515 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:22.613476    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65" event={"ID":"1a13fc89-e7b5-4bcb-b81f-d13f17498467","Type":"ContainerStarted","Data":"d2f3fd6f79356b28d92ad5c42cf3863f3ffb407ed1caf7dd7d6b0b95ddc4c231"}
Sep 09 21:58:22.614071 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:22.614046    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65" event={"ID":"1a13fc89-e7b5-4bcb-b81f-d13f17498467","Type":"ContainerStarted","Data":"9c0364a5e59784b6fdcf5de6ecae019b851d86de2998a84997fa4316a3c1217d"}
Sep 09 21:58:22.614194 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:22.614171    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:22.635357 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:22.635306    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65" podStartSLOduration=1.635291879 podStartE2EDuration="1.635291879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:58:22.632896702 +0000 UTC m=+1505.007220527" watchObservedRunningTime="2026-09-09 21:58:22.635291879 +0000 UTC m=+1505.009615702"
Sep 09 21:58:34.627652 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:34.627618    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:58:44.253112 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:44.253023    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:44.352978 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:44.352933    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:44.529138 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:44.529054    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:46.969646 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:46.969614    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:48.687448 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:48.687409    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:53.818640 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:53.818604    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:55.502945 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:55.502908    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:57.147764 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:57.147726    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:58.259345 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:58.259301    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:58.355510 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:58.355481    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:59.523767 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.523728    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"]
Sep 09 21:58:59.524272 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.524024    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw" podUID="c13a3ec0-04ae-4a63-9e31-efeb966f47af" containerName="maas-api" containerID="cri-o://0ceb6614396140cfb9a44dfb58b6a3eadaf4bf4a0107bba89cc4ca5f6359c639" gracePeriod=30
Sep 09 21:58:59.550586 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.550536    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:59.580049 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.580010    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"]
Sep 09 21:58:59.581041 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.580772    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj" podUID="8ca6d3a5-3b0a-48de-9220-114a74f6483e" containerName="payload-pre-processing" containerID="cri-o://24ba7fa3042ee17b85325c48ce01784f353e1dbd87876115e3e1d8e0d9b44daf" gracePeriod=30
Sep 09 21:58:59.650366 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.650335    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:59.752774 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.752685    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:59.826690 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.820409    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ca6d3a5-3b0a-48de-9220-114a74f6483e" containerID="24ba7fa3042ee17b85325c48ce01784f353e1dbd87876115e3e1d8e0d9b44daf" exitCode=0
Sep 09 21:58:59.826690 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.820552    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj" event={"ID":"8ca6d3a5-3b0a-48de-9220-114a74f6483e","Type":"ContainerDied","Data":"24ba7fa3042ee17b85325c48ce01784f353e1dbd87876115e3e1d8e0d9b44daf"}
Sep 09 21:58:59.826690 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.822397    2585 generic.go:356] "Generic (PLEG): container finished" podID="c13a3ec0-04ae-4a63-9e31-efeb966f47af" containerID="0ceb6614396140cfb9a44dfb58b6a3eadaf4bf4a0107bba89cc4ca5f6359c639" exitCode=0
Sep 09 21:58:59.826690 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.822431    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw" event={"ID":"c13a3ec0-04ae-4a63-9e31-efeb966f47af","Type":"ContainerDied","Data":"0ceb6614396140cfb9a44dfb58b6a3eadaf4bf4a0107bba89cc4ca5f6359c639"}
Sep 09 21:58:59.852376 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.852341    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:59.853076 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.853051    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:58:59.904814 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.904729    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:58:59.953448 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.953420    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpr7r\" (UniqueName: \"kubernetes.io/projected/c13a3ec0-04ae-4a63-9e31-efeb966f47af-kube-api-access-gpr7r\") pod \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") "
Sep 09 21:58:59.953607 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.953464    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-metrics-certs\") pod \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") "
Sep 09 21:58:59.953607 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.953480    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls\") pod \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") "
Sep 09 21:58:59.953607 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.953509    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbgl4\" (UniqueName: \"kubernetes.io/projected/8ca6d3a5-3b0a-48de-9220-114a74f6483e-kube-api-access-sbgl4\") pod \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\" (UID: \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\") "
Sep 09 21:58:59.953607 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.953533    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca6d3a5-3b0a-48de-9220-114a74f6483e-plugins-config-volume\") pod \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\" (UID: \"8ca6d3a5-3b0a-48de-9220-114a74f6483e\") "
Sep 09 21:58:59.953823 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.953665    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13a3ec0-04ae-4a63-9e31-efeb966f47af-openshift-service-ca\") pod \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\" (UID: \"c13a3ec0-04ae-4a63-9e31-efeb966f47af\") "
Sep 09 21:58:59.954010 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.953969    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca6d3a5-3b0a-48de-9220-114a74f6483e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ca6d3a5-3b0a-48de-9220-114a74f6483e" (UID: "8ca6d3a5-3b0a-48de-9220-114a74f6483e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:59.954131 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.954107    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca6d3a5-3b0a-48de-9220-114a74f6483e-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:58:59.954383 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.954351    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13a3ec0-04ae-4a63-9e31-efeb966f47af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c13a3ec0-04ae-4a63-9e31-efeb966f47af" (UID: "c13a3ec0-04ae-4a63-9e31-efeb966f47af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:59.956052 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.956022    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13a3ec0-04ae-4a63-9e31-efeb966f47af-kube-api-access-gpr7r" (OuterVolumeSpecName: "kube-api-access-gpr7r") pod "c13a3ec0-04ae-4a63-9e31-efeb966f47af" (UID: "c13a3ec0-04ae-4a63-9e31-efeb966f47af"). InnerVolumeSpecName "kube-api-access-gpr7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:59.956142 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.956099    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c13a3ec0-04ae-4a63-9e31-efeb966f47af" (UID: "c13a3ec0-04ae-4a63-9e31-efeb966f47af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:58:59.956338 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.956309    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca6d3a5-3b0a-48de-9220-114a74f6483e-kube-api-access-sbgl4" (OuterVolumeSpecName: "kube-api-access-sbgl4") pod "8ca6d3a5-3b0a-48de-9220-114a74f6483e" (UID: "8ca6d3a5-3b0a-48de-9220-114a74f6483e"). InnerVolumeSpecName "kube-api-access-sbgl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:59.956421 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:58:59.956356    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c13a3ec0-04ae-4a63-9e31-efeb966f47af" (UID: "c13a3ec0-04ae-4a63-9e31-efeb966f47af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:00.055175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.055087    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13a3ec0-04ae-4a63-9e31-efeb966f47af-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:00.055175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.055117    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpr7r\" (UniqueName: \"kubernetes.io/projected/c13a3ec0-04ae-4a63-9e31-efeb966f47af-kube-api-access-gpr7r\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:00.055175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.055129    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:00.055175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.055137    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13a3ec0-04ae-4a63-9e31-efeb966f47af-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:00.055175 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.055146    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbgl4\" (UniqueName: \"kubernetes.io/projected/8ca6d3a5-3b0a-48de-9220-114a74f6483e-kube-api-access-sbgl4\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:00.101542 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.101481    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:00.828739 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.828654    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"
Sep 09 21:59:00.828739 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.828697    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw" event={"ID":"c13a3ec0-04ae-4a63-9e31-efeb966f47af","Type":"ContainerDied","Data":"0e868b896ed02144d059318885effa81a4427d19e4dc42b2b3ff1f4631e1f643"}
Sep 09 21:59:00.829246 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.828759    2585 scope.go:117] "RemoveContainer" containerID="0ceb6614396140cfb9a44dfb58b6a3eadaf4bf4a0107bba89cc4ca5f6359c639"
Sep 09 21:59:00.830273 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.830243    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"
Sep 09 21:59:00.830273 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.830257    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj" event={"ID":"8ca6d3a5-3b0a-48de-9220-114a74f6483e","Type":"ContainerDied","Data":"ee983782dd6d40aebff19a07b1186f055988b86bc6d7fefaf232071833da4c34"}
Sep 09 21:59:00.838441 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.838425    2585 scope.go:117] "RemoveContainer" containerID="24ba7fa3042ee17b85325c48ce01784f353e1dbd87876115e3e1d8e0d9b44daf"
Sep 09 21:59:00.847112 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.847092    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"]
Sep 09 21:59:00.851394 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.851371    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-390abc-655f6476b9-7phqw"]
Sep 09 21:59:00.864996 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.864961    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"]
Sep 09 21:59:00.867007 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:00.866988    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-390abc-57cfb4f7f8-2hdvj"]
Sep 09 21:59:02.128651 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:02.128621    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ca6d3a5-3b0a-48de-9220-114a74f6483e" path="/var/lib/kubelet/pods/8ca6d3a5-3b0a-48de-9220-114a74f6483e/volumes"
Sep 09 21:59:02.129050 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:02.129021    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13a3ec0-04ae-4a63-9e31-efeb966f47af" path="/var/lib/kubelet/pods/c13a3ec0-04ae-4a63-9e31-efeb966f47af/volumes"
Sep 09 21:59:03.039512 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.039476    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:03.063617 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.063586    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"]
Sep 09 21:59:03.063869 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.063842    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg" podUID="6f3a06ef-f6bb-419c-9a38-bb1165c10dbb" containerName="payload-processing" containerID="cri-o://112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b" gracePeriod=30
Sep 09 21:59:03.073559 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.073532    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"]
Sep 09 21:59:03.073800 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.073772    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr" podUID="c0014150-3259-4647-b770-4e944b147ba3" containerName="payload-pre-processing" containerID="cri-o://6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab" gracePeriod=30
Sep 09 21:59:03.116336 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.116308    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:03.218899 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.218859    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:03.303759 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.303669    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:03.370957 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.370926    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:59:03.373514 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.373475    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:03.378529 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.378507    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:59:03.490990 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.490959    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s78qh\" (UniqueName: \"kubernetes.io/projected/c0014150-3259-4647-b770-4e944b147ba3-kube-api-access-s78qh\") pod \"c0014150-3259-4647-b770-4e944b147ba3\" (UID: \"c0014150-3259-4647-b770-4e944b147ba3\") "
Sep 09 21:59:03.491182 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.491074    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d2fd\" (UniqueName: \"kubernetes.io/projected/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-kube-api-access-4d2fd\") pod \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\" (UID: \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\") "
Sep 09 21:59:03.491182 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.491176    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-plugins-config-volume\") pod \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\" (UID: \"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb\") "
Sep 09 21:59:03.491324 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.491225    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0014150-3259-4647-b770-4e944b147ba3-plugins-config-volume\") pod \"c0014150-3259-4647-b770-4e944b147ba3\" (UID: \"c0014150-3259-4647-b770-4e944b147ba3\") "
Sep 09 21:59:03.491585 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.491558    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f3a06ef-f6bb-419c-9a38-bb1165c10dbb" (UID: "6f3a06ef-f6bb-419c-9a38-bb1165c10dbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:03.491637 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.491558    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0014150-3259-4647-b770-4e944b147ba3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0014150-3259-4647-b770-4e944b147ba3" (UID: "c0014150-3259-4647-b770-4e944b147ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:03.493252 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.493230    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0014150-3259-4647-b770-4e944b147ba3-kube-api-access-s78qh" (OuterVolumeSpecName: "kube-api-access-s78qh") pod "c0014150-3259-4647-b770-4e944b147ba3" (UID: "c0014150-3259-4647-b770-4e944b147ba3"). InnerVolumeSpecName "kube-api-access-s78qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:03.493389 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.493368    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-kube-api-access-4d2fd" (OuterVolumeSpecName: "kube-api-access-4d2fd") pod "6f3a06ef-f6bb-419c-9a38-bb1165c10dbb" (UID: "6f3a06ef-f6bb-419c-9a38-bb1165c10dbb"). InnerVolumeSpecName "kube-api-access-4d2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:03.593045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.592966    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s78qh\" (UniqueName: \"kubernetes.io/projected/c0014150-3259-4647-b770-4e944b147ba3-kube-api-access-s78qh\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:03.593045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.592994    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d2fd\" (UniqueName: \"kubernetes.io/projected/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-kube-api-access-4d2fd\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:03.593045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.593004    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:03.593045 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.593013    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0014150-3259-4647-b770-4e944b147ba3-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:03.623431 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.623403    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:03.852135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.852037    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f3a06ef-f6bb-419c-9a38-bb1165c10dbb" containerID="112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b" exitCode=0
Sep 09 21:59:03.852135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.852093    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"
Sep 09 21:59:03.852135 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.852120    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg" event={"ID":"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb","Type":"ContainerDied","Data":"112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b"}
Sep 09 21:59:03.852437 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.852226    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg" event={"ID":"6f3a06ef-f6bb-419c-9a38-bb1165c10dbb","Type":"ContainerDied","Data":"da844f2af2e1c9866782f52f220de579566991f740bda25befdf4d1970d99ee5"}
Sep 09 21:59:03.852437 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.852232    2585 scope.go:117] "RemoveContainer" containerID="112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b"
Sep 09 21:59:03.853977 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.853953    2585 generic.go:356] "Generic (PLEG): container finished" podID="c0014150-3259-4647-b770-4e944b147ba3" containerID="6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab" exitCode=0
Sep 09 21:59:03.854113 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.853995    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr" event={"ID":"c0014150-3259-4647-b770-4e944b147ba3","Type":"ContainerDied","Data":"6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab"}
Sep 09 21:59:03.854200 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.854178    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr" event={"ID":"c0014150-3259-4647-b770-4e944b147ba3","Type":"ContainerDied","Data":"b5981e81700b73c1dd593ff40ca87e09887244268f18880cf650e4322ac07635"}
Sep 09 21:59:03.854200 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.854022    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"
Sep 09 21:59:03.865048 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.864929    2585 scope.go:117] "RemoveContainer" containerID="112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b"
Sep 09 21:59:03.865367 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:59:03.865326    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b\": container with ID starting with 112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b not found: ID does not exist" containerID="112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b"
Sep 09 21:59:03.865476 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.865375    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b"} err="failed to get container status \"112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b\": rpc error: code = NotFound desc = could not find container \"112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b\": container with ID starting with 112e99bc8e5baa57036c37d68c16f44ad16254eda00705d65be6556c156cfc9b not found: ID does not exist"
Sep 09 21:59:03.865476 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.865397    2585 scope.go:117] "RemoveContainer" containerID="6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab"
Sep 09 21:59:03.876187 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.876165    2585 scope.go:117] "RemoveContainer" containerID="6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab"
Sep 09 21:59:03.876627 ip-10-0-128-94 kubenswrapper[2585]: E0909 21:59:03.876585    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab\": container with ID starting with 6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab not found: ID does not exist" containerID="6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab"
Sep 09 21:59:03.876627 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.876627    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab"} err="failed to get container status \"6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab\": rpc error: code = NotFound desc = could not find container \"6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab\": container with ID starting with 6a13a861201a529d5700f68d15a3719142c9c8957c6b325005e8c7faa6b105ab not found: ID does not exist"
Sep 09 21:59:03.879702 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.879673    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"]
Sep 09 21:59:03.884480 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.884454    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d8401-559dd65bf8-524vr"]
Sep 09 21:59:03.894341 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.894317    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"]
Sep 09 21:59:03.898063 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:03.898044    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8401-76887ccf55-vslsg"]
Sep 09 21:59:04.129739 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:04.129663    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f3a06ef-f6bb-419c-9a38-bb1165c10dbb" path="/var/lib/kubelet/pods/6f3a06ef-f6bb-419c-9a38-bb1165c10dbb/volumes"
Sep 09 21:59:04.130040 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:04.130027    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0014150-3259-4647-b770-4e944b147ba3" path="/var/lib/kubelet/pods/c0014150-3259-4647-b770-4e944b147ba3/volumes"
Sep 09 21:59:06.497190 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:06.497156    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:06.621075 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:06.621042    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:06.880255 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:06.880146    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:07.753118 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:07.753067    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:09.534102 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:09.534065    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:09.614165 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:09.614113    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:09.852634 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:09.852514    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:10.503808 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:10.503760    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:24.852539 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:24.852498    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:24.950735 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:24.950702    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:25.778490 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:25.778449    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:25.816854 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:25.816818    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"]
Sep 09 21:59:25.817246 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:25.817190    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65" podUID="1a13fc89-e7b5-4bcb-b81f-d13f17498467" containerName="payload-pre-processing" containerID="cri-o://d2f3fd6f79356b28d92ad5c42cf3863f3ffb407ed1caf7dd7d6b0b95ddc4c231" gracePeriod=30
Sep 09 21:59:25.885224 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:25.885181    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:25.978582 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:25.978550    2585 generic.go:356] "Generic (PLEG): container finished" podID="1a13fc89-e7b5-4bcb-b81f-d13f17498467" containerID="d2f3fd6f79356b28d92ad5c42cf3863f3ffb407ed1caf7dd7d6b0b95ddc4c231" exitCode=0
Sep 09 21:59:25.978755 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:25.978624    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65" event={"ID":"1a13fc89-e7b5-4bcb-b81f-d13f17498467","Type":"ContainerDied","Data":"d2f3fd6f79356b28d92ad5c42cf3863f3ffb407ed1caf7dd7d6b0b95ddc4c231"}
Sep 09 21:59:26.091732 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.091703    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:59:26.094661 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.094631    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:26.121375 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.121346    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgjrj\" (UniqueName: \"kubernetes.io/projected/1a13fc89-e7b5-4bcb-b81f-d13f17498467-kube-api-access-jgjrj\") pod \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\" (UID: \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\") "
Sep 09 21:59:26.121375 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.121384    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a13fc89-e7b5-4bcb-b81f-d13f17498467-plugins-config-volume\") pod \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\" (UID: \"1a13fc89-e7b5-4bcb-b81f-d13f17498467\") "
Sep 09 21:59:26.121937 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.121902    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a13fc89-e7b5-4bcb-b81f-d13f17498467-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a13fc89-e7b5-4bcb-b81f-d13f17498467" (UID: "1a13fc89-e7b5-4bcb-b81f-d13f17498467"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:26.122427 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.122406    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a13fc89-e7b5-4bcb-b81f-d13f17498467-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:26.123782 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.123757    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a13fc89-e7b5-4bcb-b81f-d13f17498467-kube-api-access-jgjrj" (OuterVolumeSpecName: "kube-api-access-jgjrj") pod "1a13fc89-e7b5-4bcb-b81f-d13f17498467" (UID: "1a13fc89-e7b5-4bcb-b81f-d13f17498467"). InnerVolumeSpecName "kube-api-access-jgjrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:26.224011 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.223975    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgjrj\" (UniqueName: \"kubernetes.io/projected/1a13fc89-e7b5-4bcb-b81f-d13f17498467-kube-api-access-jgjrj\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:26.235192 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.235163    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:26.617826 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.617791    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:26.985577 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.985549    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"
Sep 09 21:59:26.985963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.985553    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65" event={"ID":"1a13fc89-e7b5-4bcb-b81f-d13f17498467","Type":"ContainerDied","Data":"9c0364a5e59784b6fdcf5de6ecae019b851d86de2998a84997fa4316a3c1217d"}
Sep 09 21:59:26.985963 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:26.985692    2585 scope.go:117] "RemoveContainer" containerID="d2f3fd6f79356b28d92ad5c42cf3863f3ffb407ed1caf7dd7d6b0b95ddc4c231"
Sep 09 21:59:27.013232 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:27.013176    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"]
Sep 09 21:59:27.022976 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:27.022954    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dedad-d569b978-vnq65"]
Sep 09 21:59:27.902849 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:27.902807    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:28.129307 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:28.129272    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a13fc89-e7b5-4bcb-b81f-d13f17498467" path="/var/lib/kubelet/pods/1a13fc89-e7b5-4bcb-b81f-d13f17498467/volumes"
Sep 09 21:59:33.691756 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:33.691717    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:33.838745 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:33.838709    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:34.076065 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:34.075980    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:34.519012 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:34.518984    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:37.356556 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:37.356523    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:37.475779 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:37.475743    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:40.876852 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:40.876816    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:44.872727 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:44.872682    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:44.901342 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:44.901297    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"]
Sep 09 21:59:44.901611 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:44.901560    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2" podUID="d4035eeb-b7f2-4ee7-b2b0-e77673665b1a" containerName="payload-processing" containerID="cri-o://708f19d7574d70a84097960639e2e41a06ef9b357db0b4ee424bb0c6d409b993" gracePeriod=30
Sep 09 21:59:44.917089 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:44.917056    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"]
Sep 09 21:59:44.917414 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:44.917372    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf" podUID="489d5c38-2b28-4baa-aab6-711c20df083a" containerName="payload-pre-processing" containerID="cri-o://8ebfec0dc02963df4698a16ba3655f04499e55e057397bfd51f1948c3529691c" gracePeriod=30
Sep 09 21:59:44.960677 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:44.960642    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:45.043357 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.043311    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:45.086058 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.086027    2585 generic.go:356] "Generic (PLEG): container finished" podID="489d5c38-2b28-4baa-aab6-711c20df083a" containerID="8ebfec0dc02963df4698a16ba3655f04499e55e057397bfd51f1948c3529691c" exitCode=0
Sep 09 21:59:45.086284 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.086242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf" event={"ID":"489d5c38-2b28-4baa-aab6-711c20df083a","Type":"ContainerDied","Data":"8ebfec0dc02963df4698a16ba3655f04499e55e057397bfd51f1948c3529691c"}
Sep 09 21:59:45.088704 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.088625    2585 generic.go:356] "Generic (PLEG): container finished" podID="d4035eeb-b7f2-4ee7-b2b0-e77673665b1a" containerID="708f19d7574d70a84097960639e2e41a06ef9b357db0b4ee424bb0c6d409b993" exitCode=0
Sep 09 21:59:45.088704 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.088662    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2" event={"ID":"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a","Type":"ContainerDied","Data":"708f19d7574d70a84097960639e2e41a06ef9b357db0b4ee424bb0c6d409b993"}
Sep 09 21:59:45.113248 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.113168    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:45.186874 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.186836    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:45.209948 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.209927    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:59:45.221412 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.221386    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:59:45.299958 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.299920    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss4xz\" (UniqueName: \"kubernetes.io/projected/489d5c38-2b28-4baa-aab6-711c20df083a-kube-api-access-ss4xz\") pod \"489d5c38-2b28-4baa-aab6-711c20df083a\" (UID: \"489d5c38-2b28-4baa-aab6-711c20df083a\") "
Sep 09 21:59:45.300141 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.299967    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d5c38-2b28-4baa-aab6-711c20df083a-plugins-config-volume\") pod \"489d5c38-2b28-4baa-aab6-711c20df083a\" (UID: \"489d5c38-2b28-4baa-aab6-711c20df083a\") "
Sep 09 21:59:45.300141 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.299993    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4kvj\" (UniqueName: \"kubernetes.io/projected/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-kube-api-access-h4kvj\") pod \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\" (UID: \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\") "
Sep 09 21:59:45.300141 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.300021    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-plugins-config-volume\") pod \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\" (UID: \"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a\") "
Sep 09 21:59:45.300478 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.300443    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489d5c38-2b28-4baa-aab6-711c20df083a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "489d5c38-2b28-4baa-aab6-711c20df083a" (UID: "489d5c38-2b28-4baa-aab6-711c20df083a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:45.300559 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.300479    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4035eeb-b7f2-4ee7-b2b0-e77673665b1a" (UID: "d4035eeb-b7f2-4ee7-b2b0-e77673665b1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:45.302364 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.302339    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489d5c38-2b28-4baa-aab6-711c20df083a-kube-api-access-ss4xz" (OuterVolumeSpecName: "kube-api-access-ss4xz") pod "489d5c38-2b28-4baa-aab6-711c20df083a" (UID: "489d5c38-2b28-4baa-aab6-711c20df083a"). InnerVolumeSpecName "kube-api-access-ss4xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:45.302489 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.302472    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-kube-api-access-h4kvj" (OuterVolumeSpecName: "kube-api-access-h4kvj") pod "d4035eeb-b7f2-4ee7-b2b0-e77673665b1a" (UID: "d4035eeb-b7f2-4ee7-b2b0-e77673665b1a"). InnerVolumeSpecName "kube-api-access-h4kvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:45.400847 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.400763    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4xz\" (UniqueName: \"kubernetes.io/projected/489d5c38-2b28-4baa-aab6-711c20df083a-kube-api-access-ss4xz\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:45.400847 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.400797    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d5c38-2b28-4baa-aab6-711c20df083a-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:45.400847 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.400808    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4kvj\" (UniqueName: \"kubernetes.io/projected/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-kube-api-access-h4kvj\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:45.400847 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.400818    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 21:59:45.508340 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:45.508302    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:46.095069 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.095039    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"
Sep 09 21:59:46.095536 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.095073    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2" event={"ID":"d4035eeb-b7f2-4ee7-b2b0-e77673665b1a","Type":"ContainerDied","Data":"b3211fd0cbb06eeffa4ed52f460c5c9df8545578a4669437902718bd4443ebcb"}
Sep 09 21:59:46.095536 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.095147    2585 scope.go:117] "RemoveContainer" containerID="708f19d7574d70a84097960639e2e41a06ef9b357db0b4ee424bb0c6d409b993"
Sep 09 21:59:46.096583 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.096557    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf" event={"ID":"489d5c38-2b28-4baa-aab6-711c20df083a","Type":"ContainerDied","Data":"2ffa3cef1121cb47d107a22110ae32cebe8848bc759ba0e799a97ad03166b50e"}
Sep 09 21:59:46.096583 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.096578    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"
Sep 09 21:59:46.105386 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.105365    2585 scope.go:117] "RemoveContainer" containerID="8ebfec0dc02963df4698a16ba3655f04499e55e057397bfd51f1948c3529691c"
Sep 09 21:59:46.131268 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.131242    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"]
Sep 09 21:59:46.137167 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.137144    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d99c83-86bf9975d9-8cnl2"]
Sep 09 21:59:46.148560 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.148537    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"]
Sep 09 21:59:46.153797 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:46.153772    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d99c83-6988bf498-v2dqf"]
Sep 09 21:59:48.129038 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:48.129003    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="489d5c38-2b28-4baa-aab6-711c20df083a" path="/var/lib/kubelet/pods/489d5c38-2b28-4baa-aab6-711c20df083a/volumes"
Sep 09 21:59:48.129446 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:48.129400    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4035eeb-b7f2-4ee7-b2b0-e77673665b1a" path="/var/lib/kubelet/pods/d4035eeb-b7f2-4ee7-b2b0-e77673665b1a/volumes"
Sep 09 21:59:52.154369 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:52.154337    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:52.359110 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:52.359073    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:52.540399 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:52.540316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:53.043642 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:53.043605    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:55.019666 ip-10-0-128-94 kubenswrapper[2585]: I0909 21:59:55.019625    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:01.582595 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:01.582561    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:03.553940 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.553906    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"]
Sep 09 22:00:03.554396 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.554135    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww" podUID="030f7245-2aa2-41c4-abf7-6cbe13aa0308" containerName="maas-api" containerID="cri-o://578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1" gracePeriod=30
Sep 09 22:00:03.643011 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.642977    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"]
Sep 09 22:00:03.643256 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.643230    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9" podUID="d0435875-19c9-48be-ad5e-43e7715f2794" containerName="payload-processing" containerID="cri-o://d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b" gracePeriod=30
Sep 09 22:00:03.647116 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.647078    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"]
Sep 09 22:00:03.647508 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.647477    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2" podUID="16578d45-ca6a-4d5e-b801-4da7e08e0f16" containerName="payload-pre-processing" containerID="cri-o://6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69" gracePeriod=30
Sep 09 22:00:03.940604 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.940574    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 22:00:03.946615 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.946587    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:03.962324 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.962293    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 22:00:03.974540 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:03.974515    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 22:00:04.088853 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.088762    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls\") pod \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") "
Sep 09 22:00:04.088853 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.088815    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-metrics-certs\") pod \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") "
Sep 09 22:00:04.089090 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.088879    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0435875-19c9-48be-ad5e-43e7715f2794-plugins-config-volume\") pod \"d0435875-19c9-48be-ad5e-43e7715f2794\" (UID: \"d0435875-19c9-48be-ad5e-43e7715f2794\") "
Sep 09 22:00:04.089090 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.088907    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpvhk\" (UniqueName: \"kubernetes.io/projected/16578d45-ca6a-4d5e-b801-4da7e08e0f16-kube-api-access-qpvhk\") pod \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\" (UID: \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\") "
Sep 09 22:00:04.089090 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.088942    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76j5b\" (UniqueName: \"kubernetes.io/projected/d0435875-19c9-48be-ad5e-43e7715f2794-kube-api-access-76j5b\") pod \"d0435875-19c9-48be-ad5e-43e7715f2794\" (UID: \"d0435875-19c9-48be-ad5e-43e7715f2794\") "
Sep 09 22:00:04.089090 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.088972    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m4n7\" (UniqueName: \"kubernetes.io/projected/030f7245-2aa2-41c4-abf7-6cbe13aa0308-kube-api-access-9m4n7\") pod \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") "
Sep 09 22:00:04.089090 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.089012    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16578d45-ca6a-4d5e-b801-4da7e08e0f16-plugins-config-volume\") pod \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\" (UID: \"16578d45-ca6a-4d5e-b801-4da7e08e0f16\") "
Sep 09 22:00:04.089090 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.089077    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030f7245-2aa2-41c4-abf7-6cbe13aa0308-openshift-service-ca\") pod \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\" (UID: \"030f7245-2aa2-41c4-abf7-6cbe13aa0308\") "
Sep 09 22:00:04.089519 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.089493    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0435875-19c9-48be-ad5e-43e7715f2794-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0435875-19c9-48be-ad5e-43e7715f2794" (UID: "d0435875-19c9-48be-ad5e-43e7715f2794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:00:04.089890 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.089860    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16578d45-ca6a-4d5e-b801-4da7e08e0f16-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16578d45-ca6a-4d5e-b801-4da7e08e0f16" (UID: "16578d45-ca6a-4d5e-b801-4da7e08e0f16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:00:04.090021 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.089992    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030f7245-2aa2-41c4-abf7-6cbe13aa0308-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "030f7245-2aa2-41c4-abf7-6cbe13aa0308" (UID: "030f7245-2aa2-41c4-abf7-6cbe13aa0308"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:00:04.091385 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.091357    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030f7245-2aa2-41c4-abf7-6cbe13aa0308-kube-api-access-9m4n7" (OuterVolumeSpecName: "kube-api-access-9m4n7") pod "030f7245-2aa2-41c4-abf7-6cbe13aa0308" (UID: "030f7245-2aa2-41c4-abf7-6cbe13aa0308"). InnerVolumeSpecName "kube-api-access-9m4n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:04.091385 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.091355    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "030f7245-2aa2-41c4-abf7-6cbe13aa0308" (UID: "030f7245-2aa2-41c4-abf7-6cbe13aa0308"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:00:04.091688 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.091669    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16578d45-ca6a-4d5e-b801-4da7e08e0f16-kube-api-access-qpvhk" (OuterVolumeSpecName: "kube-api-access-qpvhk") pod "16578d45-ca6a-4d5e-b801-4da7e08e0f16" (UID: "16578d45-ca6a-4d5e-b801-4da7e08e0f16"). InnerVolumeSpecName "kube-api-access-qpvhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:04.091688 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.091669    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "030f7245-2aa2-41c4-abf7-6cbe13aa0308" (UID: "030f7245-2aa2-41c4-abf7-6cbe13aa0308"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:00:04.091782 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.091694    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0435875-19c9-48be-ad5e-43e7715f2794-kube-api-access-76j5b" (OuterVolumeSpecName: "kube-api-access-76j5b") pod "d0435875-19c9-48be-ad5e-43e7715f2794" (UID: "d0435875-19c9-48be-ad5e-43e7715f2794"). InnerVolumeSpecName "kube-api-access-76j5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:04.184920 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.184854    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:04.190182 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.190150    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0435875-19c9-48be-ad5e-43e7715f2794-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:00:04.190345 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.190184    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpvhk\" (UniqueName: \"kubernetes.io/projected/16578d45-ca6a-4d5e-b801-4da7e08e0f16-kube-api-access-qpvhk\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:00:04.190345 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.190198    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76j5b\" (UniqueName: \"kubernetes.io/projected/d0435875-19c9-48be-ad5e-43e7715f2794-kube-api-access-76j5b\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:00:04.190345 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.190244    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m4n7\" (UniqueName: \"kubernetes.io/projected/030f7245-2aa2-41c4-abf7-6cbe13aa0308-kube-api-access-9m4n7\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:00:04.190345 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.190258    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16578d45-ca6a-4d5e-b801-4da7e08e0f16-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:00:04.190345 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.190270    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030f7245-2aa2-41c4-abf7-6cbe13aa0308-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:00:04.190345 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.190282    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:00:04.190345 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.190294    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030f7245-2aa2-41c4-abf7-6cbe13aa0308-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:00:04.196589 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.196564    2585 generic.go:356] "Generic (PLEG): container finished" podID="16578d45-ca6a-4d5e-b801-4da7e08e0f16" containerID="6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69" exitCode=0
Sep 09 22:00:04.196697 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.196620    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"
Sep 09 22:00:04.196697 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.196648    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2" event={"ID":"16578d45-ca6a-4d5e-b801-4da7e08e0f16","Type":"ContainerDied","Data":"6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69"}
Sep 09 22:00:04.196799 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.196697    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2" event={"ID":"16578d45-ca6a-4d5e-b801-4da7e08e0f16","Type":"ContainerDied","Data":"f234ff071409c117d9a70c9cad96586d33e72667a67799a50d8afab1b2030245"}
Sep 09 22:00:04.196799 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.196719    2585 scope.go:117] "RemoveContainer" containerID="6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69"
Sep 09 22:00:04.198228 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.198179    2585 generic.go:356] "Generic (PLEG): container finished" podID="d0435875-19c9-48be-ad5e-43e7715f2794" containerID="d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b" exitCode=0
Sep 09 22:00:04.198324 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.198242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9" event={"ID":"d0435875-19c9-48be-ad5e-43e7715f2794","Type":"ContainerDied","Data":"d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b"}
Sep 09 22:00:04.198324 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.198292    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9" event={"ID":"d0435875-19c9-48be-ad5e-43e7715f2794","Type":"ContainerDied","Data":"2eebb08657550d102d323b262b80707aa09e280ad7a95f6578bf1c817c61347c"}
Sep 09 22:00:04.198324 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.198247    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"
Sep 09 22:00:04.199936 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.199914    2585 generic.go:356] "Generic (PLEG): container finished" podID="030f7245-2aa2-41c4-abf7-6cbe13aa0308" containerID="578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1" exitCode=0
Sep 09 22:00:04.200044 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.199956    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"
Sep 09 22:00:04.200044 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.199977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww" event={"ID":"030f7245-2aa2-41c4-abf7-6cbe13aa0308","Type":"ContainerDied","Data":"578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1"}
Sep 09 22:00:04.200044 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.200015    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww" event={"ID":"030f7245-2aa2-41c4-abf7-6cbe13aa0308","Type":"ContainerDied","Data":"c45f228341bdfbd4ae33f0230097111ce4ac1fff4b4369c7b503954a872edf83"}
Sep 09 22:00:04.206441 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.206420    2585 scope.go:117] "RemoveContainer" containerID="6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69"
Sep 09 22:00:04.206731 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:00:04.206710    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69\": container with ID starting with 6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69 not found: ID does not exist" containerID="6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69"
Sep 09 22:00:04.206820 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.206738    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69"} err="failed to get container status \"6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69\": rpc error: code = NotFound desc = could not find container \"6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69\": container with ID starting with 6bec06d4cbfda79893428a545cafc98e1a06d73fd06fe98cb9138149a5ad1e69 not found: ID does not exist"
Sep 09 22:00:04.206820 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.206754    2585 scope.go:117] "RemoveContainer" containerID="d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b"
Sep 09 22:00:04.217430 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.217387    2585 scope.go:117] "RemoveContainer" containerID="d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b"
Sep 09 22:00:04.217788 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:00:04.217764    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b\": container with ID starting with d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b not found: ID does not exist" containerID="d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b"
Sep 09 22:00:04.217872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.217798    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b"} err="failed to get container status \"d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b\": rpc error: code = NotFound desc = could not find container \"d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b\": container with ID starting with d28b29942a0fb66d92e2a7fd96f09223b2e6b94d5216b8ace2adbd9d97c0065b not found: ID does not exist"
Sep 09 22:00:04.217872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.217819    2585 scope.go:117] "RemoveContainer" containerID="578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1"
Sep 09 22:00:04.224344 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.224319    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"]
Sep 09 22:00:04.227442 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.227420    2585 scope.go:117] "RemoveContainer" containerID="578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1"
Sep 09 22:00:04.227808 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:00:04.227790    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1\": container with ID starting with 578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1 not found: ID does not exist" containerID="578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1"
Sep 09 22:00:04.227882 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.227816    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1"} err="failed to get container status \"578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1\": rpc error: code = NotFound desc = could not find container \"578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1\": container with ID starting with 578d474c64d5702240ac6df7af103c15af905c2c3508d07764251c6c1c92a5d1 not found: ID does not exist"
Sep 09 22:00:04.229484 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.229465    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9e275c-679b996fd8-6lgww"]
Sep 09 22:00:04.239962 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.239940    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"]
Sep 09 22:00:04.243260 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.243239    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e275c-64575c7d97-ttmb9"]
Sep 09 22:00:04.247777 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.247753    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:04.260053 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.255896    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"]
Sep 09 22:00:04.262740 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:04.262715    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e275c-7994c94758-9ksr2"]
Sep 09 22:00:05.167180 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:05.167138    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:06.128111 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:06.128070    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="030f7245-2aa2-41c4-abf7-6cbe13aa0308" path="/var/lib/kubelet/pods/030f7245-2aa2-41c4-abf7-6cbe13aa0308/volumes"
Sep 09 22:00:06.128640 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:06.128614    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16578d45-ca6a-4d5e-b801-4da7e08e0f16" path="/var/lib/kubelet/pods/16578d45-ca6a-4d5e-b801-4da7e08e0f16/volumes"
Sep 09 22:00:06.129019 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:06.129001    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0435875-19c9-48be-ad5e-43e7715f2794" path="/var/lib/kubelet/pods/d0435875-19c9-48be-ad5e-43e7715f2794/volumes"
Sep 09 22:00:10.559297 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:10.559184    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:10.660410 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:10.660365    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:12.400038 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:12.400005    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:14.824719 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:14.824682    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:27.038909 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:27.038876    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:33.838525 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:00:33.838486    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:14.633947 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:14.633914    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:33.235984 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:33.235945    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:34.581381 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:34.581333    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"]
Sep 09 22:01:34.601654 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:34.601624    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:40.715963 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:40.715885    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:49.602381 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:49.602345    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc" podUID="edadd607-4df0-44ea-982b-4209c2a1be30" containerName="main" containerID="cri-o://b5da73fb293bb3564828bd5928f278555c9e6348b16fd24a2c2967bfe9c3b998" gracePeriod=45
Sep 09 22:01:49.737572 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:49.737538    2585 generic.go:356] "Generic (PLEG): container finished" podID="edadd607-4df0-44ea-982b-4209c2a1be30" containerID="b5da73fb293bb3564828bd5928f278555c9e6348b16fd24a2c2967bfe9c3b998" exitCode=0
Sep 09 22:01:49.737743 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:49.737585    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc" event={"ID":"edadd607-4df0-44ea-982b-4209c2a1be30","Type":"ContainerDied","Data":"b5da73fb293bb3564828bd5928f278555c9e6348b16fd24a2c2967bfe9c3b998"}
Sep 09 22:01:49.860733 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:49.860674    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 22:01:50.009745 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.009712    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-tmp-dir\") pod \"edadd607-4df0-44ea-982b-4209c2a1be30\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") "
Sep 09 22:01:50.009745 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.009742    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-home\") pod \"edadd607-4df0-44ea-982b-4209c2a1be30\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") "
Sep 09 22:01:50.009997 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.009776    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-model-cache\") pod \"edadd607-4df0-44ea-982b-4209c2a1be30\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") "
Sep 09 22:01:50.009997 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.009802    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqr55\" (UniqueName: \"kubernetes.io/projected/edadd607-4df0-44ea-982b-4209c2a1be30-kube-api-access-rqr55\") pod \"edadd607-4df0-44ea-982b-4209c2a1be30\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") "
Sep 09 22:01:50.009997 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.009825    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edadd607-4df0-44ea-982b-4209c2a1be30-tls-certs\") pod \"edadd607-4df0-44ea-982b-4209c2a1be30\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") "
Sep 09 22:01:50.009997 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.009845    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-dshm\") pod \"edadd607-4df0-44ea-982b-4209c2a1be30\" (UID: \"edadd607-4df0-44ea-982b-4209c2a1be30\") "
Sep 09 22:01:50.010184 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.010045    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edadd607-4df0-44ea-982b-4209c2a1be30" (UID: "edadd607-4df0-44ea-982b-4209c2a1be30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:01:50.010184 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.010060    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-home" (OuterVolumeSpecName: "home") pod "edadd607-4df0-44ea-982b-4209c2a1be30" (UID: "edadd607-4df0-44ea-982b-4209c2a1be30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:01:50.010184 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.010072    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-model-cache" (OuterVolumeSpecName: "model-cache") pod "edadd607-4df0-44ea-982b-4209c2a1be30" (UID: "edadd607-4df0-44ea-982b-4209c2a1be30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:01:50.010378 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.010186    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:01:50.010378 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.010240    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:01:50.010378 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.010255    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:01:50.012256 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.012232    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-dshm" (OuterVolumeSpecName: "dshm") pod "edadd607-4df0-44ea-982b-4209c2a1be30" (UID: "edadd607-4df0-44ea-982b-4209c2a1be30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:01:50.012540 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.012467    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edadd607-4df0-44ea-982b-4209c2a1be30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edadd607-4df0-44ea-982b-4209c2a1be30" (UID: "edadd607-4df0-44ea-982b-4209c2a1be30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:01:50.012540 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.012482    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edadd607-4df0-44ea-982b-4209c2a1be30-kube-api-access-rqr55" (OuterVolumeSpecName: "kube-api-access-rqr55") pod "edadd607-4df0-44ea-982b-4209c2a1be30" (UID: "edadd607-4df0-44ea-982b-4209c2a1be30"). InnerVolumeSpecName "kube-api-access-rqr55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:01:50.111415 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.111320    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqr55\" (UniqueName: \"kubernetes.io/projected/edadd607-4df0-44ea-982b-4209c2a1be30-kube-api-access-rqr55\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:01:50.111415 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.111361    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edadd607-4df0-44ea-982b-4209c2a1be30-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:01:50.111415 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.111376    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edadd607-4df0-44ea-982b-4209c2a1be30-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:01:50.744968 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.744943    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"
Sep 09 22:01:50.745470 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.744942    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc" event={"ID":"edadd607-4df0-44ea-982b-4209c2a1be30","Type":"ContainerDied","Data":"c141482b21cdd2a6c0c4c5b6c59a1fa1e3b49114713df9ea6680c49a1cbc429f"}
Sep 09 22:01:50.745470 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.745090    2585 scope.go:117] "RemoveContainer" containerID="b5da73fb293bb3564828bd5928f278555c9e6348b16fd24a2c2967bfe9c3b998"
Sep 09 22:01:50.761186 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.761160    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"]
Sep 09 22:01:50.765541 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:50.765518    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/auth-test-model-64fec6-kserve-5d6cc4595f-m5qmc"]
Sep 09 22:01:52.128389 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:01:52.128345    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edadd607-4df0-44ea-982b-4209c2a1be30" path="/var/lib/kubelet/pods/edadd607-4df0-44ea-982b-4209c2a1be30/volumes"
Sep 09 22:02:02.429699 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:02.429660    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:09.767184 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:09.767148    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:14.780730 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:14.780691    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"]
Sep 09 22:02:14.798993 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:14.798959    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"]
Sep 09 22:02:14.858447 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:14.858408    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:25.394315 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.394258    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"]
Sep 09 22:02:25.395726 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.395384    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" podUID="c8ed369c-b75a-4cde-ba5e-9cc8c857a675" containerName="maas-api" containerID="cri-o://d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355" gracePeriod=30
Sep 09 22:02:25.433899 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.433869    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:25.464562 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.464530    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"]
Sep 09 22:02:25.464778 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.464752    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2" podUID="397ba615-e709-42f4-b6e0-f22cba40f113" containerName="payload-processing" containerID="cri-o://cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b" gracePeriod=30
Sep 09 22:02:25.471200 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.470904    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"]
Sep 09 22:02:25.471200 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.471153    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c" podUID="3674b913-41e2-484b-86a8-74bd53f0f0b8" containerName="payload-pre-processing" containerID="cri-o://ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df" gracePeriod=30
Sep 09 22:02:25.498745 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.498711    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" podUID="c8ed369c-b75a-4cde-ba5e-9cc8c857a675" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": dial tcp 10.134.0.32:8443: connect: connection refused"
Sep 09 22:02:25.539040 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.539013    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:25.765710 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.765683    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 22:02:25.768654 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.768633    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 22:02:25.783774 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.783745    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 22:02:25.869283 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869253    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674b913-41e2-484b-86a8-74bd53f0f0b8-plugins-config-volume\") pod \"3674b913-41e2-484b-86a8-74bd53f0f0b8\" (UID: \"3674b913-41e2-484b-86a8-74bd53f0f0b8\") "
Sep 09 22:02:25.869283 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869287    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-openshift-service-ca\") pod \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") "
Sep 09 22:02:25.869529 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869330    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcc2k\" (UniqueName: \"kubernetes.io/projected/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-kube-api-access-rcc2k\") pod \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") "
Sep 09 22:02:25.869529 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869361    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbtz2\" (UniqueName: \"kubernetes.io/projected/397ba615-e709-42f4-b6e0-f22cba40f113-kube-api-access-kbtz2\") pod \"397ba615-e709-42f4-b6e0-f22cba40f113\" (UID: \"397ba615-e709-42f4-b6e0-f22cba40f113\") "
Sep 09 22:02:25.869529 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869409    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls\") pod \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") "
Sep 09 22:02:25.869529 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869437    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-metrics-certs\") pod \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\" (UID: \"c8ed369c-b75a-4cde-ba5e-9cc8c857a675\") "
Sep 09 22:02:25.869742 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869539    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmwrk\" (UniqueName: \"kubernetes.io/projected/3674b913-41e2-484b-86a8-74bd53f0f0b8-kube-api-access-wmwrk\") pod \"3674b913-41e2-484b-86a8-74bd53f0f0b8\" (UID: \"3674b913-41e2-484b-86a8-74bd53f0f0b8\") "
Sep 09 22:02:25.869805 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869729    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8ed369c-b75a-4cde-ba5e-9cc8c857a675" (UID: "c8ed369c-b75a-4cde-ba5e-9cc8c857a675"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:02:25.869805 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869734    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3674b913-41e2-484b-86a8-74bd53f0f0b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3674b913-41e2-484b-86a8-74bd53f0f0b8" (UID: "3674b913-41e2-484b-86a8-74bd53f0f0b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:02:25.869805 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.869792    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397ba615-e709-42f4-b6e0-f22cba40f113-plugins-config-volume\") pod \"397ba615-e709-42f4-b6e0-f22cba40f113\" (UID: \"397ba615-e709-42f4-b6e0-f22cba40f113\") "
Sep 09 22:02:25.870495 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.870466    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397ba615-e709-42f4-b6e0-f22cba40f113-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "397ba615-e709-42f4-b6e0-f22cba40f113" (UID: "397ba615-e709-42f4-b6e0-f22cba40f113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:02:25.870659 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.870544    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674b913-41e2-484b-86a8-74bd53f0f0b8-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:25.870659 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.870566    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:25.871933 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.871907    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8ed369c-b75a-4cde-ba5e-9cc8c857a675" (UID: "c8ed369c-b75a-4cde-ba5e-9cc8c857a675"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:02:25.872041 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.871963    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c8ed369c-b75a-4cde-ba5e-9cc8c857a675" (UID: "c8ed369c-b75a-4cde-ba5e-9cc8c857a675"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:02:25.872041 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.871987    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3674b913-41e2-484b-86a8-74bd53f0f0b8-kube-api-access-wmwrk" (OuterVolumeSpecName: "kube-api-access-wmwrk") pod "3674b913-41e2-484b-86a8-74bd53f0f0b8" (UID: "3674b913-41e2-484b-86a8-74bd53f0f0b8"). InnerVolumeSpecName "kube-api-access-wmwrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:25.872298 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.872278    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-kube-api-access-rcc2k" (OuterVolumeSpecName: "kube-api-access-rcc2k") pod "c8ed369c-b75a-4cde-ba5e-9cc8c857a675" (UID: "c8ed369c-b75a-4cde-ba5e-9cc8c857a675"). InnerVolumeSpecName "kube-api-access-rcc2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:25.872389 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.872318    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397ba615-e709-42f4-b6e0-f22cba40f113-kube-api-access-kbtz2" (OuterVolumeSpecName: "kube-api-access-kbtz2") pod "397ba615-e709-42f4-b6e0-f22cba40f113" (UID: "397ba615-e709-42f4-b6e0-f22cba40f113"). InnerVolumeSpecName "kube-api-access-kbtz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:25.919710 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.919671    2585 generic.go:356] "Generic (PLEG): container finished" podID="3674b913-41e2-484b-86a8-74bd53f0f0b8" containerID="ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df" exitCode=0
Sep 09 22:02:25.919875 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.919714    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"
Sep 09 22:02:25.919875 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.919761    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c" event={"ID":"3674b913-41e2-484b-86a8-74bd53f0f0b8","Type":"ContainerDied","Data":"ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df"}
Sep 09 22:02:25.919875 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.919828    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c" event={"ID":"3674b913-41e2-484b-86a8-74bd53f0f0b8","Type":"ContainerDied","Data":"830b10bf1fb327517cd576a78b4af0b518336326fb79346087b48ada409a435f"}
Sep 09 22:02:25.919875 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.919860    2585 scope.go:117] "RemoveContainer" containerID="ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df"
Sep 09 22:02:25.921283 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.921259    2585 generic.go:356] "Generic (PLEG): container finished" podID="397ba615-e709-42f4-b6e0-f22cba40f113" containerID="cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b" exitCode=0
Sep 09 22:02:25.921420 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.921319    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"
Sep 09 22:02:25.921420 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.921354    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2" event={"ID":"397ba615-e709-42f4-b6e0-f22cba40f113","Type":"ContainerDied","Data":"cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b"}
Sep 09 22:02:25.921420 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.921415    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2" event={"ID":"397ba615-e709-42f4-b6e0-f22cba40f113","Type":"ContainerDied","Data":"ba0de8c6f27b25ef6de7371601109c3d9e2750c76fce63228869f98eb708700c"}
Sep 09 22:02:25.922866 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.922844    2585 generic.go:356] "Generic (PLEG): container finished" podID="c8ed369c-b75a-4cde-ba5e-9cc8c857a675" containerID="d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355" exitCode=0
Sep 09 22:02:25.922981 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.922886    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"
Sep 09 22:02:25.922981 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.922927    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" event={"ID":"c8ed369c-b75a-4cde-ba5e-9cc8c857a675","Type":"ContainerDied","Data":"d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355"}
Sep 09 22:02:25.923078 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.922985    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9" event={"ID":"c8ed369c-b75a-4cde-ba5e-9cc8c857a675","Type":"ContainerDied","Data":"b39e913a07add516d0a3380d64834627366b10d767bfbdccf67b0d23cffc5ecd"}
Sep 09 22:02:25.932471 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.932400    2585 scope.go:117] "RemoveContainer" containerID="ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df"
Sep 09 22:02:25.934439 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:02:25.934415    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df\": container with ID starting with ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df not found: ID does not exist" containerID="ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df"
Sep 09 22:02:25.934574 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.934536    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df"} err="failed to get container status \"ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df\": rpc error: code = NotFound desc = could not find container \"ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df\": container with ID starting with ce0120551f4a3821d3606123c413cb8d89a14222c4703a265f5698c2aad6d5df not found: ID does not exist"
Sep 09 22:02:25.934574 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.934575    2585 scope.go:117] "RemoveContainer" containerID="cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b"
Sep 09 22:02:25.944259 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.944242    2585 scope.go:117] "RemoveContainer" containerID="cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b"
Sep 09 22:02:25.944514 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:02:25.944496    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b\": container with ID starting with cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b not found: ID does not exist" containerID="cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b"
Sep 09 22:02:25.944572 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.944521    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b"} err="failed to get container status \"cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b\": rpc error: code = NotFound desc = could not find container \"cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b\": container with ID starting with cedd30ab972d2f83d5e7000eb5458083b88ca9dfdf4135070f1074d344648e3b not found: ID does not exist"
Sep 09 22:02:25.944572 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.944539    2585 scope.go:117] "RemoveContainer" containerID="d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355"
Sep 09 22:02:25.952811 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.952787    2585 scope.go:117] "RemoveContainer" containerID="d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355"
Sep 09 22:02:25.953075 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:02:25.953052    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355\": container with ID starting with d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355 not found: ID does not exist" containerID="d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355"
Sep 09 22:02:25.953126 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.953086    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355"} err="failed to get container status \"d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355\": rpc error: code = NotFound desc = could not find container \"d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355\": container with ID starting with d0fe806877a1ffbb1d072e05159e8d5fd0620d5b335dacb25bf779f439ca5355 not found: ID does not exist"
Sep 09 22:02:25.971404 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.971385    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcc2k\" (UniqueName: \"kubernetes.io/projected/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-kube-api-access-rcc2k\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:25.971404 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.971405    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbtz2\" (UniqueName: \"kubernetes.io/projected/397ba615-e709-42f4-b6e0-f22cba40f113-kube-api-access-kbtz2\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:25.971536 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.971415    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:25.971536 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.971426    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ed369c-b75a-4cde-ba5e-9cc8c857a675-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:25.971536 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.971434    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwrk\" (UniqueName: \"kubernetes.io/projected/3674b913-41e2-484b-86a8-74bd53f0f0b8-kube-api-access-wmwrk\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:25.971536 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.971441    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397ba615-e709-42f4-b6e0-f22cba40f113-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:25.974314 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.974291    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"]
Sep 09 22:02:25.986961 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:25.986941    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fa986-7767558c7c-vstt2"]
Sep 09 22:02:26.020786 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.020756    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"]
Sep 09 22:02:26.027067 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.027044    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fa986-5869fb8cd-tzl7c"]
Sep 09 22:02:26.051025 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.050995    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"]
Sep 09 22:02:26.056591 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.056567    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fa986-6bbb477894-89nk9"]
Sep 09 22:02:26.127747 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.127712    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3674b913-41e2-484b-86a8-74bd53f0f0b8" path="/var/lib/kubelet/pods/3674b913-41e2-484b-86a8-74bd53f0f0b8/volumes"
Sep 09 22:02:26.128087 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.128074    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="397ba615-e709-42f4-b6e0-f22cba40f113" path="/var/lib/kubelet/pods/397ba615-e709-42f4-b6e0-f22cba40f113/volumes"
Sep 09 22:02:26.128438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.128425    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ed369c-b75a-4cde-ba5e-9cc8c857a675" path="/var/lib/kubelet/pods/c8ed369c-b75a-4cde-ba5e-9cc8c857a675/volumes"
Sep 09 22:02:26.184942 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.184915    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:26.284632 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:26.284547    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:29.801730 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:29.801688    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf" podUID="6ab68ead-a5d8-4271-9e66-6723e3b926fe" containerName="main" containerID="cri-o://a1fd68d284db4ea83f4548e93e18bb549e11a77dfbffd7c9baf12d5987a02d23" gracePeriod=45
Sep 09 22:02:29.820337 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:29.820303    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8" podUID="92ceeeb1-782c-4638-a172-fff61580c2d3" containerName="main" containerID="cri-o://e49248493aef68adc7424aa713be51318d472d6f6c1a84e60bbb7b9559070ac1" gracePeriod=45
Sep 09 22:02:29.834150 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:29.834115    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8" podUID="92ceeeb1-782c-4638-a172-fff61580c2d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 09 22:02:29.950061 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:29.950029    2585 generic.go:356] "Generic (PLEG): container finished" podID="6ab68ead-a5d8-4271-9e66-6723e3b926fe" containerID="a1fd68d284db4ea83f4548e93e18bb549e11a77dfbffd7c9baf12d5987a02d23" exitCode=0
Sep 09 22:02:29.950264 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:29.950188    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf" event={"ID":"6ab68ead-a5d8-4271-9e66-6723e3b926fe","Type":"ContainerDied","Data":"a1fd68d284db4ea83f4548e93e18bb549e11a77dfbffd7c9baf12d5987a02d23"}
Sep 09 22:02:29.951982 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:29.951951    2585 generic.go:356] "Generic (PLEG): container finished" podID="92ceeeb1-782c-4638-a172-fff61580c2d3" containerID="e49248493aef68adc7424aa713be51318d472d6f6c1a84e60bbb7b9559070ac1" exitCode=0
Sep 09 22:02:29.952082 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:29.952001    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8" event={"ID":"92ceeeb1-782c-4638-a172-fff61580c2d3","Type":"ContainerDied","Data":"e49248493aef68adc7424aa713be51318d472d6f6c1a84e60bbb7b9559070ac1"}
Sep 09 22:02:30.083993 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.083967    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 22:02:30.105796 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.105770    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 22:02:30.215270 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215236    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-home\") pod \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") "
Sep 09 22:02:30.215270 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215271    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-dshm\") pod \"92ceeeb1-782c-4638-a172-fff61580c2d3\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") "
Sep 09 22:02:30.215534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215295    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tn75\" (UniqueName: \"kubernetes.io/projected/6ab68ead-a5d8-4271-9e66-6723e3b926fe-kube-api-access-4tn75\") pod \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") "
Sep 09 22:02:30.215534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215312    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fzkj\" (UniqueName: \"kubernetes.io/projected/92ceeeb1-782c-4638-a172-fff61580c2d3-kube-api-access-2fzkj\") pod \"92ceeeb1-782c-4638-a172-fff61580c2d3\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") "
Sep 09 22:02:30.215534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215345    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ceeeb1-782c-4638-a172-fff61580c2d3-tls-certs\") pod \"92ceeeb1-782c-4638-a172-fff61580c2d3\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") "
Sep 09 22:02:30.215534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215382    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-model-cache\") pod \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") "
Sep 09 22:02:30.215534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215442    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tls-certs\") pod \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") "
Sep 09 22:02:30.215534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215466    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-tmp-dir\") pod \"92ceeeb1-782c-4638-a172-fff61580c2d3\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") "
Sep 09 22:02:30.215534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215490    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-model-cache\") pod \"92ceeeb1-782c-4638-a172-fff61580c2d3\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") "
Sep 09 22:02:30.215534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215518    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-home" (OuterVolumeSpecName: "home") pod "6ab68ead-a5d8-4271-9e66-6723e3b926fe" (UID: "6ab68ead-a5d8-4271-9e66-6723e3b926fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.215936 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215579    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-dshm\") pod \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") "
Sep 09 22:02:30.215936 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215628    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tmp-dir\") pod \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\" (UID: \"6ab68ead-a5d8-4271-9e66-6723e3b926fe\") "
Sep 09 22:02:30.215936 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215653    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-home\") pod \"92ceeeb1-782c-4638-a172-fff61580c2d3\" (UID: \"92ceeeb1-782c-4638-a172-fff61580c2d3\") "
Sep 09 22:02:30.215936 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215841    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "92ceeeb1-782c-4638-a172-fff61580c2d3" (UID: "92ceeeb1-782c-4638-a172-fff61580c2d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.216138 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.215978    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92ceeeb1-782c-4638-a172-fff61580c2d3" (UID: "92ceeeb1-782c-4638-a172-fff61580c2d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.216233 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.216197    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.216303 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.216237    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.216360 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.216295    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ab68ead-a5d8-4271-9e66-6723e3b926fe" (UID: "6ab68ead-a5d8-4271-9e66-6723e3b926fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.216483 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.216460    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-home" (OuterVolumeSpecName: "home") pod "92ceeeb1-782c-4638-a172-fff61580c2d3" (UID: "92ceeeb1-782c-4638-a172-fff61580c2d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.216698 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.216678    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ab68ead-a5d8-4271-9e66-6723e3b926fe" (UID: "6ab68ead-a5d8-4271-9e66-6723e3b926fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.218402 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.218368    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ab68ead-a5d8-4271-9e66-6723e3b926fe" (UID: "6ab68ead-a5d8-4271-9e66-6723e3b926fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:02:30.218566 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.218525    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-dshm" (OuterVolumeSpecName: "dshm") pod "92ceeeb1-782c-4638-a172-fff61580c2d3" (UID: "92ceeeb1-782c-4638-a172-fff61580c2d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.218640 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.218533    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab68ead-a5d8-4271-9e66-6723e3b926fe-kube-api-access-4tn75" (OuterVolumeSpecName: "kube-api-access-4tn75") pod "6ab68ead-a5d8-4271-9e66-6723e3b926fe" (UID: "6ab68ead-a5d8-4271-9e66-6723e3b926fe"). InnerVolumeSpecName "kube-api-access-4tn75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:30.218867 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.218845    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ceeeb1-782c-4638-a172-fff61580c2d3-kube-api-access-2fzkj" (OuterVolumeSpecName: "kube-api-access-2fzkj") pod "92ceeeb1-782c-4638-a172-fff61580c2d3" (UID: "92ceeeb1-782c-4638-a172-fff61580c2d3"). InnerVolumeSpecName "kube-api-access-2fzkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:30.218948 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.218931    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-dshm" (OuterVolumeSpecName: "dshm") pod "6ab68ead-a5d8-4271-9e66-6723e3b926fe" (UID: "6ab68ead-a5d8-4271-9e66-6723e3b926fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.219108 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.219091    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ceeeb1-782c-4638-a172-fff61580c2d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92ceeeb1-782c-4638-a172-fff61580c2d3" (UID: "92ceeeb1-782c-4638-a172-fff61580c2d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:02:30.317674 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317560    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.317674 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317617    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.317674 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317631    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.317674 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317644    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tn75\" (UniqueName: \"kubernetes.io/projected/6ab68ead-a5d8-4271-9e66-6723e3b926fe-kube-api-access-4tn75\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.317674 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317661    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fzkj\" (UniqueName: \"kubernetes.io/projected/92ceeeb1-782c-4638-a172-fff61580c2d3-kube-api-access-2fzkj\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.317674 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317675    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ceeeb1-782c-4638-a172-fff61580c2d3-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.317674 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317688    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.318011 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317701    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab68ead-a5d8-4271-9e66-6723e3b926fe-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.318011 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317713    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ceeeb1-782c-4638-a172-fff61580c2d3-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.318011 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.317725    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab68ead-a5d8-4271-9e66-6723e3b926fe-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.958641 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.958613    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"
Sep 09 22:02:30.959124 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.958647    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf" event={"ID":"6ab68ead-a5d8-4271-9e66-6723e3b926fe","Type":"ContainerDied","Data":"dda5c886890953ca8fddc94347cd53c380948cf200cb58f98e95734025568fd0"}
Sep 09 22:02:30.959124 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.958705    2585 scope.go:117] "RemoveContainer" containerID="a1fd68d284db4ea83f4548e93e18bb549e11a77dfbffd7c9baf12d5987a02d23"
Sep 09 22:02:30.960368 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.960343    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8" event={"ID":"92ceeeb1-782c-4638-a172-fff61580c2d3","Type":"ContainerDied","Data":"d86b2d4fc70445658a69d05e55405098f98dc0ff64c3fec2c7ce4ce8f794d3f2"}
Sep 09 22:02:30.960368 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.960367    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"
Sep 09 22:02:30.969091 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.969072    2585 scope.go:117] "RemoveContainer" containerID="e49248493aef68adc7424aa713be51318d472d6f6c1a84e60bbb7b9559070ac1"
Sep 09 22:02:30.982860 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.982836    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"]
Sep 09 22:02:30.987063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.987040    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/unconfigured-w4-3fa986-kserve-647c87cfd9-dt5h8"]
Sep 09 22:02:30.996250 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:30.996230    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"]
Sep 09 22:02:31.000645 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:31.000618    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/premium-w4-3fa986-kserve-f6c6754cf-zgckf"]
Sep 09 22:02:32.104675 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:32.104644    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:32.127890 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:32.127858    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab68ead-a5d8-4271-9e66-6723e3b926fe" path="/var/lib/kubelet/pods/6ab68ead-a5d8-4271-9e66-6723e3b926fe/volumes"
Sep 09 22:02:32.128262 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:32.128248    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ceeeb1-782c-4638-a172-fff61580c2d3" path="/var/lib/kubelet/pods/92ceeeb1-782c-4638-a172-fff61580c2d3/volumes"
Sep 09 22:02:32.206024 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:32.205991    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:32.480265 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:32.480232    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:35.550719 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:35.550686    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:51.547803 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:51.547768    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:51.646826 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:51.646796    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:52.556499 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:02:52.556461    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:03.140795 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:03.140761    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:04.368190 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:04.368154    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:07.478727 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:07.478690    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:13.598625 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598538    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"]
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598881    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489d5c38-2b28-4baa-aab6-711c20df083a" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598891    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="489d5c38-2b28-4baa-aab6-711c20df083a" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598920    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ceeeb1-782c-4638-a172-fff61580c2d3" containerName="main"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598925    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ceeeb1-782c-4638-a172-fff61580c2d3" containerName="main"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598934    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab68ead-a5d8-4271-9e66-6723e3b926fe" containerName="main"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598940    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab68ead-a5d8-4271-9e66-6723e3b926fe" containerName="main"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598947    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0435875-19c9-48be-ad5e-43e7715f2794" containerName="payload-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598952    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0435875-19c9-48be-ad5e-43e7715f2794" containerName="payload-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598958    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edadd607-4df0-44ea-982b-4209c2a1be30" containerName="main"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598963    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="edadd607-4df0-44ea-982b-4209c2a1be30" containerName="main"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598970    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca6d3a5-3b0a-48de-9220-114a74f6483e" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598975    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca6d3a5-3b0a-48de-9220-114a74f6483e" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598981    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16578d45-ca6a-4d5e-b801-4da7e08e0f16" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598986    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="16578d45-ca6a-4d5e-b801-4da7e08e0f16" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598993    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030f7245-2aa2-41c4-abf7-6cbe13aa0308" containerName="maas-api"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.598997    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="030f7245-2aa2-41c4-abf7-6cbe13aa0308" containerName="maas-api"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599004    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f3a06ef-f6bb-419c-9a38-bb1165c10dbb" containerName="payload-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599010    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f3a06ef-f6bb-419c-9a38-bb1165c10dbb" containerName="payload-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599017    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0014150-3259-4647-b770-4e944b147ba3" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599025    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0014150-3259-4647-b770-4e944b147ba3" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599036    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a13fc89-e7b5-4bcb-b81f-d13f17498467" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599043    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a13fc89-e7b5-4bcb-b81f-d13f17498467" containerName="payload-pre-processing"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599059    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13a3ec0-04ae-4a63-9e31-efeb966f47af" containerName="maas-api"
Sep 09 22:03:13.599063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599067    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13a3ec0-04ae-4a63-9e31-efeb966f47af" containerName="maas-api"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599078    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397ba615-e709-42f4-b6e0-f22cba40f113" containerName="payload-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599086    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="397ba615-e709-42f4-b6e0-f22cba40f113" containerName="payload-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599096    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4035eeb-b7f2-4ee7-b2b0-e77673665b1a" containerName="payload-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599103    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4035eeb-b7f2-4ee7-b2b0-e77673665b1a" containerName="payload-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599116    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ed369c-b75a-4cde-ba5e-9cc8c857a675" containerName="maas-api"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599123    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ed369c-b75a-4cde-ba5e-9cc8c857a675" containerName="maas-api"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599134    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3674b913-41e2-484b-86a8-74bd53f0f0b8" containerName="payload-pre-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599142    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3674b913-41e2-484b-86a8-74bd53f0f0b8" containerName="payload-pre-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599257    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab68ead-a5d8-4271-9e66-6723e3b926fe" containerName="main"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599268    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13a3ec0-04ae-4a63-9e31-efeb966f47af" containerName="maas-api"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599276    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3674b913-41e2-484b-86a8-74bd53f0f0b8" containerName="payload-pre-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599282    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4035eeb-b7f2-4ee7-b2b0-e77673665b1a" containerName="payload-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599286    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="489d5c38-2b28-4baa-aab6-711c20df083a" containerName="payload-pre-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599291    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="397ba615-e709-42f4-b6e0-f22cba40f113" containerName="payload-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599297    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ed369c-b75a-4cde-ba5e-9cc8c857a675" containerName="maas-api"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599301    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0014150-3259-4647-b770-4e944b147ba3" containerName="payload-pre-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599307    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a13fc89-e7b5-4bcb-b81f-d13f17498467" containerName="payload-pre-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599311    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ceeeb1-782c-4638-a172-fff61580c2d3" containerName="main"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599317    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="edadd607-4df0-44ea-982b-4209c2a1be30" containerName="main"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599324    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca6d3a5-3b0a-48de-9220-114a74f6483e" containerName="payload-pre-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599329    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f3a06ef-f6bb-419c-9a38-bb1165c10dbb" containerName="payload-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599334    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0435875-19c9-48be-ad5e-43e7715f2794" containerName="payload-processing"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599338    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="030f7245-2aa2-41c4-abf7-6cbe13aa0308" containerName="maas-api"
Sep 09 22:03:13.599872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.599344    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="16578d45-ca6a-4d5e-b801-4da7e08e0f16" containerName="payload-pre-processing"
Sep 09 22:03:13.602538 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.602516    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.605869 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.605844    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"openshift-service-ca.crt\""
Sep 09 22:03:13.605869 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.605865    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"rate-test-model-2d3bc8-kserve-self-signed-certs\""
Sep 09 22:03:13.606067 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.605865    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"default-dockercfg-s9tcs\""
Sep 09 22:03:13.606067 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.605900    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"kube-root-ca.crt\""
Sep 09 22:03:13.611049 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.611030    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"]
Sep 09 22:03:13.615742 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.615713    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-model-cache\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.615876 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.615755    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-home\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.615876 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.615805    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tmp-dir\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.616007 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.615871    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5jb\" (UniqueName: \"kubernetes.io/projected/fefabc2d-3bb4-4b3d-a699-5756f0841e00-kube-api-access-sm5jb\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.616007 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.615906    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-dshm\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.616125 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.616009    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tls-certs\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.651571 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.651539    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:13.716943 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.716905    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tls-certs\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.717154 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.716959    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-model-cache\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.717154 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.716987    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-home\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.717154 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.717006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tmp-dir\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.717154 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.717031    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5jb\" (UniqueName: \"kubernetes.io/projected/fefabc2d-3bb4-4b3d-a699-5756f0841e00-kube-api-access-sm5jb\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.717154 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.717056    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-dshm\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.717485 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.717444    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-home\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.717545 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.717483    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tmp-dir\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.717649 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.717629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-model-cache\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.719540 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.719512    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-dshm\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.719665 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.719591    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tls-certs\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.726574 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.726552    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5jb\" (UniqueName: \"kubernetes.io/projected/fefabc2d-3bb4-4b3d-a699-5756f0841e00-kube-api-access-sm5jb\") pod \"rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:13.914484 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:13.914449    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:14.185954 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:14.185925    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"]
Sep 09 22:03:14.188432 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:03:14.188406    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfefabc2d_3bb4_4b3d_a699_5756f0841e00.slice/crio-41b8a8a162ce12977f35a72af7ded14572a9e9b658aa9596febfafa97f53b3e8 WatchSource:0}: Error finding container 41b8a8a162ce12977f35a72af7ded14572a9e9b658aa9596febfafa97f53b3e8: Status 404 returned error can't find the container with id 41b8a8a162ce12977f35a72af7ded14572a9e9b658aa9596febfafa97f53b3e8
Sep 09 22:03:15.183825 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:15.183786    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5" event={"ID":"fefabc2d-3bb4-4b3d-a699-5756f0841e00","Type":"ContainerStarted","Data":"ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72"}
Sep 09 22:03:15.184274 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:15.183834    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5" event={"ID":"fefabc2d-3bb4-4b3d-a699-5756f0841e00","Type":"ContainerStarted","Data":"41b8a8a162ce12977f35a72af7ded14572a9e9b658aa9596febfafa97f53b3e8"}
Sep 09 22:03:15.201896 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:15.201845    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5" podStartSLOduration=2.201829398 podStartE2EDuration="2.201829398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:03:15.20049025 +0000 UTC m=+1797.574814074" watchObservedRunningTime="2026-09-09 22:03:15.201829398 +0000 UTC m=+1797.576153222"
Sep 09 22:03:18.156421 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:18.156388    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 22:03:18.157600 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:18.157580    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 22:03:19.359819 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:19.359780    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:19.426421 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:19.426386    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:23.915080 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:23.915039    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:23.915582 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:23.915107    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:23.928140 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:23.928114    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:24.236128 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:24.236039    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:03:53.402324 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.402286    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"]
Sep 09 22:03:53.405711 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.405695    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.409097 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.409067    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"default-dockercfg-m9z8s\""
Sep 09 22:03:53.409097 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.409083    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"openshift-service-ca.crt\""
Sep 09 22:03:53.409320 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.409111    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"rate-test-model-64fec6-kserve-self-signed-certs\""
Sep 09 22:03:53.409320 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.409083    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"kube-root-ca.crt\""
Sep 09 22:03:53.415710 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.415689    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"]
Sep 09 22:03:53.498503 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.498472    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:53.587807 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.587774    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvzl\" (UniqueName: \"kubernetes.io/projected/99fae541-8f33-423f-9b87-426334637966-kube-api-access-zfvzl\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.587990 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.587830    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-tmp-dir\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.587990 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.587887    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-model-cache\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.587990 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.587901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-home\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.587990 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.587916    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-dshm\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.588144 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.588007    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fae541-8f33-423f-9b87-426334637966-tls-certs\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689011 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.688915    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fae541-8f33-423f-9b87-426334637966-tls-certs\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689011 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.688975    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvzl\" (UniqueName: \"kubernetes.io/projected/99fae541-8f33-423f-9b87-426334637966-kube-api-access-zfvzl\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689011 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.689013    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-tmp-dir\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689325 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.689045    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-model-cache\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689325 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.689059    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-home\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689325 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.689075    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-dshm\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689510 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.689489    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-model-cache\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689572 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.689543    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-tmp-dir\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.689643 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.689621    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-home\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.691540 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.691521    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-dshm\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.691735 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.691719    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fae541-8f33-423f-9b87-426334637966-tls-certs\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.697821 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.697799    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvzl\" (UniqueName: \"kubernetes.io/projected/99fae541-8f33-423f-9b87-426334637966-kube-api-access-zfvzl\") pod \"rate-test-model-64fec6-kserve-689b9787-42lst\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.715782 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.715747    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:03:53.990268 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:53.990156    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"]
Sep 09 22:03:53.993973 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:03:53.993944    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99fae541_8f33_423f_9b87_426334637966.slice/crio-7e5e09b6ecf55886263da96cf215fbbe90f85e8bbf88ca1d9afd28ecee5280ae WatchSource:0}: Error finding container 7e5e09b6ecf55886263da96cf215fbbe90f85e8bbf88ca1d9afd28ecee5280ae: Status 404 returned error can't find the container with id 7e5e09b6ecf55886263da96cf215fbbe90f85e8bbf88ca1d9afd28ecee5280ae
Sep 09 22:03:54.370448 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:54.370357    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst" event={"ID":"99fae541-8f33-423f-9b87-426334637966","Type":"ContainerStarted","Data":"4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080"}
Sep 09 22:03:54.370448 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:54.370398    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst" event={"ID":"99fae541-8f33-423f-9b87-426334637966","Type":"ContainerStarted","Data":"7e5e09b6ecf55886263da96cf215fbbe90f85e8bbf88ca1d9afd28ecee5280ae"}
Sep 09 22:03:54.386447 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:03:54.386384    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst" podStartSLOduration=1.3863647000000001 podStartE2EDuration="1.3863647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:03:54.385938129 +0000 UTC m=+1836.760261954" watchObservedRunningTime="2026-09-09 22:03:54.3863647 +0000 UTC m=+1836.760688525"
Sep 09 22:04:03.716790 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:03.716749    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:04:03.717412 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:03.716808    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:04:03.729933 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:03.729906    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:04:04.433663 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:04.433630    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:04:33.878595 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:33.878562    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"]
Sep 09 22:04:33.900994 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:33.900963    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:34.819925 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:34.819888    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"]
Sep 09 22:04:35.639649 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:35.639612    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:36.030421 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:36.030393    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:38.383902 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:38.383869    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:38.534838 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:38.534810    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:39.609115 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.609029    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"]
Sep 09 22:04:39.613127 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.613102    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.615456 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.615429    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-100032-serving-cert\""
Sep 09 22:04:39.624795 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.624757    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"]
Sep 09 22:04:39.637533 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.637489    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:39.741656 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.741617    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:39.745824 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.745799    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-metrics-certs\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.745948 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.745834    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppwsl\" (UniqueName: \"kubernetes.io/projected/61fabda6-be1f-45c1-b47c-303aa486533a-kube-api-access-ppwsl\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.745948 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.745921    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fabda6-be1f-45c1-b47c-303aa486533a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.746056 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.745987    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.846911 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.846876    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppwsl\" (UniqueName: \"kubernetes.io/projected/61fabda6-be1f-45c1-b47c-303aa486533a-kube-api-access-ppwsl\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.847128 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.846966    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fabda6-be1f-45c1-b47c-303aa486533a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.847128 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.847014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.847128 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.847085    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-metrics-certs\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.847325 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:04:39.847183    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-serving-cert: secret "maas-api-e2e-ipp-a-100032-serving-cert" not found
Sep 09 22:04:39.847325 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:04:39.847289    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls podName:61fabda6-be1f-45c1-b47c-303aa486533a nodeName:}" failed. No retries permitted until 2026-09-09 22:04:40.347265956 +0000 UTC m=+1882.721589772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls") pod "maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk" (UID: "61fabda6-be1f-45c1-b47c-303aa486533a") : secret "maas-api-e2e-ipp-a-100032-serving-cert" not found
Sep 09 22:04:39.847812 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.847779    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fabda6-be1f-45c1-b47c-303aa486533a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.849731 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.849711    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-metrics-certs\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.856436 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.856413    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppwsl\" (UniqueName: \"kubernetes.io/projected/61fabda6-be1f-45c1-b47c-303aa486533a-kube-api-access-ppwsl\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:39.964006 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:39.963968    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:40.138867 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.138829    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:40.234529 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.234447    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:40.241710 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.241679    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"]
Sep 09 22:04:40.245418 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.245402    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:40.247768 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.247745    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-100032\""
Sep 09 22:04:40.247898 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.247813    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-100032-dockercfg-nqrs7\""
Sep 09 22:04:40.257076 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.257054    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"]
Sep 09 22:04:40.269886 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.269858    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"]
Sep 09 22:04:40.273067 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.273049    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:40.290568 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.290542    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"]
Sep 09 22:04:40.352548 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.352508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhqq\" (UniqueName: \"kubernetes.io/projected/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-kube-api-access-pnhqq\") pod \"payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb\" (UID: \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:40.352724 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.352566    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb\" (UID: \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:40.352724 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.352649    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72ac338-3209-4f96-bc8d-9015bfa0071b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42\" (UID: \"b72ac338-3209-4f96-bc8d-9015bfa0071b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:40.352724 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.352678    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgq74\" (UniqueName: \"kubernetes.io/projected/b72ac338-3209-4f96-bc8d-9015bfa0071b-kube-api-access-qgq74\") pod \"payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42\" (UID: \"b72ac338-3209-4f96-bc8d-9015bfa0071b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:40.352888 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.352870    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:40.356622 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.356582    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:40.453987 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.453944    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhqq\" (UniqueName: \"kubernetes.io/projected/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-kube-api-access-pnhqq\") pod \"payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb\" (UID: \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:40.453987 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.454003    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb\" (UID: \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:40.454328 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.454043    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72ac338-3209-4f96-bc8d-9015bfa0071b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42\" (UID: \"b72ac338-3209-4f96-bc8d-9015bfa0071b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:40.454328 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.454067    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgq74\" (UniqueName: \"kubernetes.io/projected/b72ac338-3209-4f96-bc8d-9015bfa0071b-kube-api-access-qgq74\") pod \"payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42\" (UID: \"b72ac338-3209-4f96-bc8d-9015bfa0071b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:40.454769 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.454748    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb\" (UID: \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:40.454810 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.454787    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72ac338-3209-4f96-bc8d-9015bfa0071b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42\" (UID: \"b72ac338-3209-4f96-bc8d-9015bfa0071b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:40.463792 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.463764    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgq74\" (UniqueName: \"kubernetes.io/projected/b72ac338-3209-4f96-bc8d-9015bfa0071b-kube-api-access-qgq74\") pod \"payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42\" (UID: \"b72ac338-3209-4f96-bc8d-9015bfa0071b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:40.464069 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.464053    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhqq\" (UniqueName: \"kubernetes.io/projected/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-kube-api-access-pnhqq\") pod \"payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb\" (UID: \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:40.527320 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.527228    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:40.557264 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.557235    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:40.587569 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.587526    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:40.807621 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.807597    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"]
Sep 09 22:04:40.810406 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:04:40.810377    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fabda6_be1f_45c1_b47c_303aa486533a.slice/crio-a9aafad66d194861408ebff4a917d82f1b0a7443b7cd18ec688f1c0f56fd61e2 WatchSource:0}: Error finding container a9aafad66d194861408ebff4a917d82f1b0a7443b7cd18ec688f1c0f56fd61e2: Status 404 returned error can't find the container with id a9aafad66d194861408ebff4a917d82f1b0a7443b7cd18ec688f1c0f56fd61e2
Sep 09 22:04:40.812138 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.812117    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 22:04:40.846275 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.846250    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"]
Sep 09 22:04:40.848637 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:04:40.848617    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81e0b8b_8b5c_433a_bf2a_865c5ebedab3.slice/crio-f2aa1b143b547fd3125723c8083c6536557db05cb256ac8b7c44254c36f12a87 WatchSource:0}: Error finding container f2aa1b143b547fd3125723c8083c6536557db05cb256ac8b7c44254c36f12a87: Status 404 returned error can't find the container with id f2aa1b143b547fd3125723c8083c6536557db05cb256ac8b7c44254c36f12a87
Sep 09 22:04:40.881771 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:40.881748    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"]
Sep 09 22:04:40.884032 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:04:40.884006    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72ac338_3209_4f96_bc8d_9015bfa0071b.slice/crio-f877cbb7a4a109aaa44f0392052a0773ee3a4ec8c6847d0dc65c5cd16d6bee00 WatchSource:0}: Error finding container f877cbb7a4a109aaa44f0392052a0773ee3a4ec8c6847d0dc65c5cd16d6bee00: Status 404 returned error can't find the container with id f877cbb7a4a109aaa44f0392052a0773ee3a4ec8c6847d0dc65c5cd16d6bee00
Sep 09 22:04:41.607410 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.607309    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42" event={"ID":"b72ac338-3209-4f96-bc8d-9015bfa0071b","Type":"ContainerStarted","Data":"e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f"}
Sep 09 22:04:41.607410 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.607380    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42" event={"ID":"b72ac338-3209-4f96-bc8d-9015bfa0071b","Type":"ContainerStarted","Data":"f877cbb7a4a109aaa44f0392052a0773ee3a4ec8c6847d0dc65c5cd16d6bee00"}
Sep 09 22:04:41.607799 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.607464    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:04:41.608989 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.608963    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb" event={"ID":"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3","Type":"ContainerStarted","Data":"613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc"}
Sep 09 22:04:41.609120 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.609013    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:41.609120 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.609037    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb" event={"ID":"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3","Type":"ContainerStarted","Data":"f2aa1b143b547fd3125723c8083c6536557db05cb256ac8b7c44254c36f12a87"}
Sep 09 22:04:41.610529 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.610503    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk" event={"ID":"61fabda6-be1f-45c1-b47c-303aa486533a","Type":"ContainerStarted","Data":"769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9"}
Sep 09 22:04:41.610642 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.610550    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:41.610642 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.610569    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk" event={"ID":"61fabda6-be1f-45c1-b47c-303aa486533a","Type":"ContainerStarted","Data":"a9aafad66d194861408ebff4a917d82f1b0a7443b7cd18ec688f1c0f56fd61e2"}
Sep 09 22:04:41.627043 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.626995    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42" podStartSLOduration=1.626978468 podStartE2EDuration="1.626978468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:04:41.625904913 +0000 UTC m=+1884.000228765" watchObservedRunningTime="2026-09-09 22:04:41.626978468 +0000 UTC m=+1884.001302297"
Sep 09 22:04:41.646067 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.646010    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb" podStartSLOduration=1.645991198 podStartE2EDuration="1.645991198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:04:41.644724133 +0000 UTC m=+1884.019047957" watchObservedRunningTime="2026-09-09 22:04:41.645991198 +0000 UTC m=+1884.020315023"
Sep 09 22:04:41.664944 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:41.664747    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk" podStartSLOduration=2.158030973 podStartE2EDuration="2.664730625s" totalImagesPullingTime="506.699652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:04:41.663842291 +0000 UTC m=+1884.038166115" watchObservedRunningTime="2026-09-09 22:04:41.664730625 +0000 UTC m=+1884.039054450"
Sep 09 22:04:47.622036 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:47.621997    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:04:48.899142 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:48.899092    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5" podUID="fefabc2d-3bb4-4b3d-a699-5756f0841e00" containerName="main" containerID="cri-o://ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72" gracePeriod=45
Sep 09 22:04:49.148507 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.148476    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:04:49.248038 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248002    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tls-certs\") pod \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") "
Sep 09 22:04:49.248259 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248054    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-home\") pod \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") "
Sep 09 22:04:49.248259 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248083    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tmp-dir\") pod \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") "
Sep 09 22:04:49.248259 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248122    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-model-cache\") pod \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") "
Sep 09 22:04:49.248259 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248157    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm5jb\" (UniqueName: \"kubernetes.io/projected/fefabc2d-3bb4-4b3d-a699-5756f0841e00-kube-api-access-sm5jb\") pod \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") "
Sep 09 22:04:49.248259 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248186    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-dshm\") pod \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\" (UID: \"fefabc2d-3bb4-4b3d-a699-5756f0841e00\") "
Sep 09 22:04:49.248547 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248397    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-home" (OuterVolumeSpecName: "home") pod "fefabc2d-3bb4-4b3d-a699-5756f0841e00" (UID: "fefabc2d-3bb4-4b3d-a699-5756f0841e00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:49.248547 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248422    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fefabc2d-3bb4-4b3d-a699-5756f0841e00" (UID: "fefabc2d-3bb4-4b3d-a699-5756f0841e00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:49.248547 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248443    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-model-cache" (OuterVolumeSpecName: "model-cache") pod "fefabc2d-3bb4-4b3d-a699-5756f0841e00" (UID: "fefabc2d-3bb4-4b3d-a699-5756f0841e00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:49.248703 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248608    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:49.248703 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248639    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:49.248703 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.248654    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:49.250621 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.250587    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fefabc2d-3bb4-4b3d-a699-5756f0841e00" (UID: "fefabc2d-3bb4-4b3d-a699-5756f0841e00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:04:49.250621 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.250600    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-dshm" (OuterVolumeSpecName: "dshm") pod "fefabc2d-3bb4-4b3d-a699-5756f0841e00" (UID: "fefabc2d-3bb4-4b3d-a699-5756f0841e00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:49.250751 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.250638    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefabc2d-3bb4-4b3d-a699-5756f0841e00-kube-api-access-sm5jb" (OuterVolumeSpecName: "kube-api-access-sm5jb") pod "fefabc2d-3bb4-4b3d-a699-5756f0841e00" (UID: "fefabc2d-3bb4-4b3d-a699-5756f0841e00"). InnerVolumeSpecName "kube-api-access-sm5jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:04:49.350043 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.350000    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefabc2d-3bb4-4b3d-a699-5756f0841e00-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:49.350043 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.350032    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm5jb\" (UniqueName: \"kubernetes.io/projected/fefabc2d-3bb4-4b3d-a699-5756f0841e00-kube-api-access-sm5jb\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:49.350043 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.350044    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefabc2d-3bb4-4b3d-a699-5756f0841e00-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:49.656455 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.656419    2585 generic.go:356] "Generic (PLEG): container finished" podID="fefabc2d-3bb4-4b3d-a699-5756f0841e00" containerID="ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72" exitCode=0
Sep 09 22:04:49.656644 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.656464    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"
Sep 09 22:04:49.656644 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.656509    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5" event={"ID":"fefabc2d-3bb4-4b3d-a699-5756f0841e00","Type":"ContainerDied","Data":"ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72"}
Sep 09 22:04:49.656644 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.656553    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5" event={"ID":"fefabc2d-3bb4-4b3d-a699-5756f0841e00","Type":"ContainerDied","Data":"41b8a8a162ce12977f35a72af7ded14572a9e9b658aa9596febfafa97f53b3e8"}
Sep 09 22:04:49.656644 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.656576    2585 scope.go:117] "RemoveContainer" containerID="ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72"
Sep 09 22:04:49.666681 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.666663    2585 scope.go:117] "RemoveContainer" containerID="ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72"
Sep 09 22:04:49.666976 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:04:49.666956    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72\": container with ID starting with ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72 not found: ID does not exist" containerID="ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72"
Sep 09 22:04:49.667030 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.666985    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72"} err="failed to get container status \"ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72\": rpc error: code = NotFound desc = could not find container \"ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72\": container with ID starting with ad436be503d00e3ad8a1abe654d0c0fada21440e68d49f06d372d32443e06e72 not found: ID does not exist"
Sep 09 22:04:49.677418 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.677394    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"]
Sep 09 22:04:49.681794 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.681772    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/rate-test-model-2d3bc8-kserve-574f449bc9-c8kn5"]
Sep 09 22:04:49.840116 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:49.840071    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst" podUID="99fae541-8f33-423f-9b87-426334637966" containerName="main" containerID="cri-o://4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080" gracePeriod=45
Sep 09 22:04:50.085058 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.085033    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:04:50.129872 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.129831    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefabc2d-3bb4-4b3d-a699-5756f0841e00" path="/var/lib/kubelet/pods/fefabc2d-3bb4-4b3d-a699-5756f0841e00/volumes"
Sep 09 22:04:50.156965 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.156936    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fae541-8f33-423f-9b87-426334637966-tls-certs\") pod \"99fae541-8f33-423f-9b87-426334637966\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") "
Sep 09 22:04:50.156965 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.156965    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-model-cache\") pod \"99fae541-8f33-423f-9b87-426334637966\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") "
Sep 09 22:04:50.157166 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157005    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-home\") pod \"99fae541-8f33-423f-9b87-426334637966\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") "
Sep 09 22:04:50.157166 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157042    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfvzl\" (UniqueName: \"kubernetes.io/projected/99fae541-8f33-423f-9b87-426334637966-kube-api-access-zfvzl\") pod \"99fae541-8f33-423f-9b87-426334637966\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") "
Sep 09 22:04:50.157166 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157066    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-tmp-dir\") pod \"99fae541-8f33-423f-9b87-426334637966\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") "
Sep 09 22:04:50.157166 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157123    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-dshm\") pod \"99fae541-8f33-423f-9b87-426334637966\" (UID: \"99fae541-8f33-423f-9b87-426334637966\") "
Sep 09 22:04:50.157423 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157260    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-model-cache" (OuterVolumeSpecName: "model-cache") pod "99fae541-8f33-423f-9b87-426334637966" (UID: "99fae541-8f33-423f-9b87-426334637966"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:50.157423 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157298    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-home" (OuterVolumeSpecName: "home") pod "99fae541-8f33-423f-9b87-426334637966" (UID: "99fae541-8f33-423f-9b87-426334637966"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:50.157423 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157373    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99fae541-8f33-423f-9b87-426334637966" (UID: "99fae541-8f33-423f-9b87-426334637966"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:50.157569 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157493    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:50.157569 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157513    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:50.157569 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.157527    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:50.159372 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.159350    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99fae541-8f33-423f-9b87-426334637966-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99fae541-8f33-423f-9b87-426334637966" (UID: "99fae541-8f33-423f-9b87-426334637966"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:04:50.159475 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.159401    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-dshm" (OuterVolumeSpecName: "dshm") pod "99fae541-8f33-423f-9b87-426334637966" (UID: "99fae541-8f33-423f-9b87-426334637966"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:50.159510 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.159465    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fae541-8f33-423f-9b87-426334637966-kube-api-access-zfvzl" (OuterVolumeSpecName: "kube-api-access-zfvzl") pod "99fae541-8f33-423f-9b87-426334637966" (UID: "99fae541-8f33-423f-9b87-426334637966"). InnerVolumeSpecName "kube-api-access-zfvzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:04:50.258630 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.258589    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fae541-8f33-423f-9b87-426334637966-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:50.258630 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.258624    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvzl\" (UniqueName: \"kubernetes.io/projected/99fae541-8f33-423f-9b87-426334637966-kube-api-access-zfvzl\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:50.258630 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.258636    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fae541-8f33-423f-9b87-426334637966-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:04:50.664277 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.664244    2585 generic.go:356] "Generic (PLEG): container finished" podID="99fae541-8f33-423f-9b87-426334637966" containerID="4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080" exitCode=0
Sep 09 22:04:50.664452 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.664289    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"
Sep 09 22:04:50.664452 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.664324    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst" event={"ID":"99fae541-8f33-423f-9b87-426334637966","Type":"ContainerDied","Data":"4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080"}
Sep 09 22:04:50.664452 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.664372    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst" event={"ID":"99fae541-8f33-423f-9b87-426334637966","Type":"ContainerDied","Data":"7e5e09b6ecf55886263da96cf215fbbe90f85e8bbf88ca1d9afd28ecee5280ae"}
Sep 09 22:04:50.664452 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.664397    2585 scope.go:117] "RemoveContainer" containerID="4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080"
Sep 09 22:04:50.675016 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.674996    2585 scope.go:117] "RemoveContainer" containerID="4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080"
Sep 09 22:04:50.675376 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:04:50.675356    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080\": container with ID starting with 4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080 not found: ID does not exist" containerID="4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080"
Sep 09 22:04:50.675436 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.675385    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080"} err="failed to get container status \"4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080\": rpc error: code = NotFound desc = could not find container \"4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080\": container with ID starting with 4294a10c80728fd89140d48d0955064966139fc909a5ff2717484707b4799080 not found: ID does not exist"
Sep 09 22:04:50.685720 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.685698    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"]
Sep 09 22:04:50.689584 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:50.689565    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/rate-test-model-64fec6-kserve-689b9787-42lst"]
Sep 09 22:04:52.128242 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:52.128187    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99fae541-8f33-423f-9b87-426334637966" path="/var/lib/kubelet/pods/99fae541-8f33-423f-9b87-426334637966/volumes"
Sep 09 22:04:52.618871 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:52.618843    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:04:52.619090 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:04:52.618922    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:05:02.143769 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:02.143727    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:03.404268 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:03.404180    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:04.133994 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:04.133957    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:04.426698 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:04.426635    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:04.721177 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:04.721083    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:05.844110 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:05.844075    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:07.864906 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:07.864842    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:07.979328 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:07.979294    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:08.701723 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.701674    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"]
Sep 09 22:05:08.702265 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.702242    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefabc2d-3bb4-4b3d-a699-5756f0841e00" containerName="main"
Sep 09 22:05:08.702429 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.702413    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefabc2d-3bb4-4b3d-a699-5756f0841e00" containerName="main"
Sep 09 22:05:08.702541 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.702517    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99fae541-8f33-423f-9b87-426334637966" containerName="main"
Sep 09 22:05:08.702541 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.702541    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="99fae541-8f33-423f-9b87-426334637966" containerName="main"
Sep 09 22:05:08.702685 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.702662    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefabc2d-3bb4-4b3d-a699-5756f0841e00" containerName="main"
Sep 09 22:05:08.702685 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.702675    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="99fae541-8f33-423f-9b87-426334637966" containerName="main"
Sep 09 22:05:08.706850 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.706829    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.709277 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.709245    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-855e86-serving-cert\""
Sep 09 22:05:08.718723 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.718642    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"]
Sep 09 22:05:08.721484 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.721447    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfb2665-88ba-472e-87b9-797dbebe862e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.721631 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.721603    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-metrics-certs\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.721718 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.721654    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qf8\" (UniqueName: \"kubernetes.io/projected/1cfb2665-88ba-472e-87b9-797dbebe862e-kube-api-access-75qf8\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.721718 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.721690    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.745330 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.745290    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:08.822288 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.822251    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-metrics-certs\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.822288 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.822300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qf8\" (UniqueName: \"kubernetes.io/projected/1cfb2665-88ba-472e-87b9-797dbebe862e-kube-api-access-75qf8\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.822554 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.822336    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.822554 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.822396    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfb2665-88ba-472e-87b9-797dbebe862e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.822554 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:05:08.822529    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-serving-cert: secret "maas-api-e2e-ipp-b-855e86-serving-cert" not found
Sep 09 22:05:08.822720 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:05:08.822610    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls podName:1cfb2665-88ba-472e-87b9-797dbebe862e nodeName:}" failed. No retries permitted until 2026-09-09 22:05:09.322590241 +0000 UTC m=+1911.696914048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls") pod "maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87" (UID: "1cfb2665-88ba-472e-87b9-797dbebe862e") : secret "maas-api-e2e-ipp-b-855e86-serving-cert" not found
Sep 09 22:05:08.823116 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.823091    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfb2665-88ba-472e-87b9-797dbebe862e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.825700 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.825636    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-metrics-certs\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.834644 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.834593    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qf8\" (UniqueName: \"kubernetes.io/projected/1cfb2665-88ba-472e-87b9-797dbebe862e-kube-api-access-75qf8\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:08.845337 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:08.845304    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:09.185185 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.185135    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:09.314109 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.314078    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:09.328248 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.328184    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:09.331007 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.330975    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:09.411496 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.411445    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:09.448683 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.448600    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"]
Sep 09 22:05:09.452075 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.452058    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:09.454452 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.454424    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-855e86\""
Sep 09 22:05:09.454585 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.454552    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-855e86-dockercfg-bs4d9\""
Sep 09 22:05:09.462380 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.462353    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"]
Sep 09 22:05:09.467601 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.467579    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"]
Sep 09 22:05:09.470851 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.470834    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:09.484543 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.484517    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"]
Sep 09 22:05:09.530394 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.530361    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wg9k\" (UniqueName: \"kubernetes.io/projected/40919460-caca-4712-9133-5dc9144188bd-kube-api-access-8wg9k\") pod \"payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc\" (UID: \"40919460-caca-4712-9133-5dc9144188bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:09.530394 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.530399    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40919460-caca-4712-9133-5dc9144188bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc\" (UID: \"40919460-caca-4712-9133-5dc9144188bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:09.530654 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.530441    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e7cd80-92e7-4865-adf5-49663deb038d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh\" (UID: \"81e7cd80-92e7-4865-adf5-49663deb038d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:09.530654 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.530583    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w974h\" (UniqueName: \"kubernetes.io/projected/81e7cd80-92e7-4865-adf5-49663deb038d-kube-api-access-w974h\") pod \"payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh\" (UID: \"81e7cd80-92e7-4865-adf5-49663deb038d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:09.621461 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.621426    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:09.632193 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.632162    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e7cd80-92e7-4865-adf5-49663deb038d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh\" (UID: \"81e7cd80-92e7-4865-adf5-49663deb038d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:09.632342 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.632268    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w974h\" (UniqueName: \"kubernetes.io/projected/81e7cd80-92e7-4865-adf5-49663deb038d-kube-api-access-w974h\") pod \"payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh\" (UID: \"81e7cd80-92e7-4865-adf5-49663deb038d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:09.632412 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.632340    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wg9k\" (UniqueName: \"kubernetes.io/projected/40919460-caca-4712-9133-5dc9144188bd-kube-api-access-8wg9k\") pod \"payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc\" (UID: \"40919460-caca-4712-9133-5dc9144188bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:09.632412 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.632363    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40919460-caca-4712-9133-5dc9144188bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc\" (UID: \"40919460-caca-4712-9133-5dc9144188bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:09.633075 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.633046    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e7cd80-92e7-4865-adf5-49663deb038d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh\" (UID: \"81e7cd80-92e7-4865-adf5-49663deb038d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:09.633272 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.633250    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40919460-caca-4712-9133-5dc9144188bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc\" (UID: \"40919460-caca-4712-9133-5dc9144188bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:09.640507 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.640481    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wg9k\" (UniqueName: \"kubernetes.io/projected/40919460-caca-4712-9133-5dc9144188bd-kube-api-access-8wg9k\") pod \"payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc\" (UID: \"40919460-caca-4712-9133-5dc9144188bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:09.640615 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.640593    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w974h\" (UniqueName: \"kubernetes.io/projected/81e7cd80-92e7-4865-adf5-49663deb038d-kube-api-access-w974h\") pod \"payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh\" (UID: \"81e7cd80-92e7-4865-adf5-49663deb038d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:09.764977 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.764884    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:09.786944 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.786920    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:09.899477 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:09.899434    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"]
Sep 09 22:05:09.900752 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:05:09.900714    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfb2665_88ba_472e_87b9_797dbebe862e.slice/crio-158d567fb6e696ec5bb867f1ce0c1b1d913172bc428b7edbb0d75ea4c126d7c4 WatchSource:0}: Error finding container 158d567fb6e696ec5bb867f1ce0c1b1d913172bc428b7edbb0d75ea4c126d7c4: Status 404 returned error can't find the container with id 158d567fb6e696ec5bb867f1ce0c1b1d913172bc428b7edbb0d75ea4c126d7c4
Sep 09 22:05:10.039096 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.038983    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"]
Sep 09 22:05:10.041274 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:05:10.041243    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40919460_caca_4712_9133_5dc9144188bd.slice/crio-4e57815a582c4f383dd91178f8fdb3fa9fc5f91effd6cbbe2f4e80124aacf6e7 WatchSource:0}: Error finding container 4e57815a582c4f383dd91178f8fdb3fa9fc5f91effd6cbbe2f4e80124aacf6e7: Status 404 returned error can't find the container with id 4e57815a582c4f383dd91178f8fdb3fa9fc5f91effd6cbbe2f4e80124aacf6e7
Sep 09 22:05:10.070031 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.070004    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"]
Sep 09 22:05:10.076440 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:05:10.076413    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e7cd80_92e7_4865_adf5_49663deb038d.slice/crio-5007c5d8b42e25ed29da15e62bf022d37b4edf38e800f6b2a3e14769bde8db84 WatchSource:0}: Error finding container 5007c5d8b42e25ed29da15e62bf022d37b4edf38e800f6b2a3e14769bde8db84: Status 404 returned error can't find the container with id 5007c5d8b42e25ed29da15e62bf022d37b4edf38e800f6b2a3e14769bde8db84
Sep 09 22:05:10.768556 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.768516    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87" event={"ID":"1cfb2665-88ba-472e-87b9-797dbebe862e","Type":"ContainerStarted","Data":"ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255"}
Sep 09 22:05:10.769179 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.768580    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87" event={"ID":"1cfb2665-88ba-472e-87b9-797dbebe862e","Type":"ContainerStarted","Data":"158d567fb6e696ec5bb867f1ce0c1b1d913172bc428b7edbb0d75ea4c126d7c4"}
Sep 09 22:05:10.769179 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.768648    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:10.770270 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.770238    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh" event={"ID":"81e7cd80-92e7-4865-adf5-49663deb038d","Type":"ContainerStarted","Data":"54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f"}
Sep 09 22:05:10.770407 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.770289    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:10.770407 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.770308    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh" event={"ID":"81e7cd80-92e7-4865-adf5-49663deb038d","Type":"ContainerStarted","Data":"5007c5d8b42e25ed29da15e62bf022d37b4edf38e800f6b2a3e14769bde8db84"}
Sep 09 22:05:10.771892 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.771871    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc" event={"ID":"40919460-caca-4712-9133-5dc9144188bd","Type":"ContainerStarted","Data":"5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e"}
Sep 09 22:05:10.771979 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.771908    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc" event={"ID":"40919460-caca-4712-9133-5dc9144188bd","Type":"ContainerStarted","Data":"4e57815a582c4f383dd91178f8fdb3fa9fc5f91effd6cbbe2f4e80124aacf6e7"}
Sep 09 22:05:10.772035 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.772023    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:10.786079 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.786036    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87" podStartSLOduration=2.44513133 podStartE2EDuration="2.786016771s" totalImagesPullingTime="340.885441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:10.785575073 +0000 UTC m=+1913.159898896" watchObservedRunningTime="2026-09-09 22:05:10.786016771 +0000 UTC m=+1913.160340599"
Sep 09 22:05:10.809722 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.805457    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc" podStartSLOduration=1.805437035 podStartE2EDuration="1.805437035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:10.801396087 +0000 UTC m=+1913.175719903" watchObservedRunningTime="2026-09-09 22:05:10.805437035 +0000 UTC m=+1913.179760859"
Sep 09 22:05:10.818289 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:10.818231    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh" podStartSLOduration=1.8182008299999999 podStartE2EDuration="1.81820083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:10.817154821 +0000 UTC m=+1913.191478646" watchObservedRunningTime="2026-09-09 22:05:10.81820083 +0000 UTC m=+1913.192524654"
Sep 09 22:05:16.783630 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:16.783595    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:05:21.779421 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:21.779387    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:05:21.779914 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:21.779666    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:05:32.674519 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:32.674484    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:34.525771 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:34.525729    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:38.017172 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:38.017142    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc_40919460-caca-4712-9133-5dc9144188bd/payload-processing/0.log"
Sep 09 22:05:44.102108 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:05:44.102068    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:24.220329 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:24.220202    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42_b72ac338-3209-4f96-bc8d-9015bfa0071b/payload-processing/0.log"
Sep 09 22:06:24.485008 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:24.484932    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc_40919460-caca-4712-9133-5dc9144188bd/payload-processing/0.log"
Sep 09 22:06:24.788142 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:24.788062    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:25.095885 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:25.095797    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:25.446565 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:25.446526    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:28.084983 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:28.084948    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:28.180315 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:28.180284    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:29.417437 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:29.417394    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:29.545757 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:29.545722    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:29.666016 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:29.665979    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:29.974093 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:29.974065    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:30.078522 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:30.078485    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:53.122773 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:53.122742    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:59.865397 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:59.865353    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:59.952178 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:06:59.952146    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:00.074945 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:00.074906    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:00.143857 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:00.143786    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:00.492698 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:00.492663    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:06.129359 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:06.129326    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:06.228547 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:06.228517    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:06.391439 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:06.391359    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:10.399156 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:10.399115    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:25.716685 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.716651    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"]
Sep 09 22:07:25.717258 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.716872    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk" podUID="61fabda6-be1f-45c1-b47c-303aa486533a" containerName="maas-api" containerID="cri-o://769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9" gracePeriod=30
Sep 09 22:07:25.749650 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.749620    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:25.774571 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.774539    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"]
Sep 09 22:07:25.774823 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.774794    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42" podUID="b72ac338-3209-4f96-bc8d-9015bfa0071b" containerName="payload-processing" containerID="cri-o://e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f" gracePeriod=30
Sep 09 22:07:25.781992 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.781949    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"]
Sep 09 22:07:25.782190 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.782164    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb" podUID="b81e0b8b-8b5c-433a-bf2a-865c5ebedab3" containerName="payload-pre-processing" containerID="cri-o://613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc" gracePeriod=30
Sep 09 22:07:25.822482 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.822450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:25.924943 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:25.924322    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.023605 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.023573    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.038400 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.038365    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:07:26.086402 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.086378    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:07:26.091107 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.091082    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:07:26.137804 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.137778    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgq74\" (UniqueName: \"kubernetes.io/projected/b72ac338-3209-4f96-bc8d-9015bfa0071b-kube-api-access-qgq74\") pod \"b72ac338-3209-4f96-bc8d-9015bfa0071b\" (UID: \"b72ac338-3209-4f96-bc8d-9015bfa0071b\") "
Sep 09 22:07:26.137967 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.137834    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppwsl\" (UniqueName: \"kubernetes.io/projected/61fabda6-be1f-45c1-b47c-303aa486533a-kube-api-access-ppwsl\") pod \"61fabda6-be1f-45c1-b47c-303aa486533a\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") "
Sep 09 22:07:26.137967 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.137856    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnhqq\" (UniqueName: \"kubernetes.io/projected/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-kube-api-access-pnhqq\") pod \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\" (UID: \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\") "
Sep 09 22:07:26.137967 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.137878    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72ac338-3209-4f96-bc8d-9015bfa0071b-plugins-config-volume\") pod \"b72ac338-3209-4f96-bc8d-9015bfa0071b\" (UID: \"b72ac338-3209-4f96-bc8d-9015bfa0071b\") "
Sep 09 22:07:26.137967 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.137917    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-metrics-certs\") pod \"61fabda6-be1f-45c1-b47c-303aa486533a\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") "
Sep 09 22:07:26.137967 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.137963    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fabda6-be1f-45c1-b47c-303aa486533a-openshift-service-ca\") pod \"61fabda6-be1f-45c1-b47c-303aa486533a\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") "
Sep 09 22:07:26.138252 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.138003    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-plugins-config-volume\") pod \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\" (UID: \"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3\") "
Sep 09 22:07:26.138252 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.138043    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls\") pod \"61fabda6-be1f-45c1-b47c-303aa486533a\" (UID: \"61fabda6-be1f-45c1-b47c-303aa486533a\") "
Sep 09 22:07:26.138369 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.138334    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72ac338-3209-4f96-bc8d-9015bfa0071b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b72ac338-3209-4f96-bc8d-9015bfa0071b" (UID: "b72ac338-3209-4f96-bc8d-9015bfa0071b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:26.138637 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.138563    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72ac338-3209-4f96-bc8d-9015bfa0071b-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.138721 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.138643    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b81e0b8b-8b5c-433a-bf2a-865c5ebedab3" (UID: "b81e0b8b-8b5c-433a-bf2a-865c5ebedab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:26.139026 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.138989    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fabda6-be1f-45c1-b47c-303aa486533a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61fabda6-be1f-45c1-b47c-303aa486533a" (UID: "61fabda6-be1f-45c1-b47c-303aa486533a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:26.140397 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.140375    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fabda6-be1f-45c1-b47c-303aa486533a-kube-api-access-ppwsl" (OuterVolumeSpecName: "kube-api-access-ppwsl") pod "61fabda6-be1f-45c1-b47c-303aa486533a" (UID: "61fabda6-be1f-45c1-b47c-303aa486533a"). InnerVolumeSpecName "kube-api-access-ppwsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:26.140764 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.140730    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72ac338-3209-4f96-bc8d-9015bfa0071b-kube-api-access-qgq74" (OuterVolumeSpecName: "kube-api-access-qgq74") pod "b72ac338-3209-4f96-bc8d-9015bfa0071b" (UID: "b72ac338-3209-4f96-bc8d-9015bfa0071b"). InnerVolumeSpecName "kube-api-access-qgq74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:26.140854 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.140793    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-kube-api-access-pnhqq" (OuterVolumeSpecName: "kube-api-access-pnhqq") pod "b81e0b8b-8b5c-433a-bf2a-865c5ebedab3" (UID: "b81e0b8b-8b5c-433a-bf2a-865c5ebedab3"). InnerVolumeSpecName "kube-api-access-pnhqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:26.140913 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.140880    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "61fabda6-be1f-45c1-b47c-303aa486533a" (UID: "61fabda6-be1f-45c1-b47c-303aa486533a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:07:26.140964 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.140951    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61fabda6-be1f-45c1-b47c-303aa486533a" (UID: "61fabda6-be1f-45c1-b47c-303aa486533a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:07:26.240044 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.240006    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.240044 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.240041    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.240044 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.240054    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgq74\" (UniqueName: \"kubernetes.io/projected/b72ac338-3209-4f96-bc8d-9015bfa0071b-kube-api-access-qgq74\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.240346 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.240063    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppwsl\" (UniqueName: \"kubernetes.io/projected/61fabda6-be1f-45c1-b47c-303aa486533a-kube-api-access-ppwsl\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.240346 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.240072    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnhqq\" (UniqueName: \"kubernetes.io/projected/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3-kube-api-access-pnhqq\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.240346 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.240080    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fabda6-be1f-45c1-b47c-303aa486533a-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.240346 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.240088    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fabda6-be1f-45c1-b47c-303aa486533a-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.431776 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.431734    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.444045 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.444014    2585 generic.go:356] "Generic (PLEG): container finished" podID="b72ac338-3209-4f96-bc8d-9015bfa0071b" containerID="e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f" exitCode=0
Sep 09 22:07:26.444192 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.444074    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"
Sep 09 22:07:26.444192 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.444103    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42" event={"ID":"b72ac338-3209-4f96-bc8d-9015bfa0071b","Type":"ContainerDied","Data":"e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f"}
Sep 09 22:07:26.444192 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.444182    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42" event={"ID":"b72ac338-3209-4f96-bc8d-9015bfa0071b","Type":"ContainerDied","Data":"f877cbb7a4a109aaa44f0392052a0773ee3a4ec8c6847d0dc65c5cd16d6bee00"}
Sep 09 22:07:26.444337 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.444227    2585 scope.go:117] "RemoveContainer" containerID="e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f"
Sep 09 22:07:26.445565 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.445546    2585 generic.go:356] "Generic (PLEG): container finished" podID="b81e0b8b-8b5c-433a-bf2a-865c5ebedab3" containerID="613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc" exitCode=0
Sep 09 22:07:26.445667 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.445609    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"
Sep 09 22:07:26.445667 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.445626    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb" event={"ID":"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3","Type":"ContainerDied","Data":"613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc"}
Sep 09 22:07:26.445775 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.445666    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb" event={"ID":"b81e0b8b-8b5c-433a-bf2a-865c5ebedab3","Type":"ContainerDied","Data":"f2aa1b143b547fd3125723c8083c6536557db05cb256ac8b7c44254c36f12a87"}
Sep 09 22:07:26.447193 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.447167    2585 generic.go:356] "Generic (PLEG): container finished" podID="61fabda6-be1f-45c1-b47c-303aa486533a" containerID="769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9" exitCode=0
Sep 09 22:07:26.447332 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.447198    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk" event={"ID":"61fabda6-be1f-45c1-b47c-303aa486533a","Type":"ContainerDied","Data":"769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9"}
Sep 09 22:07:26.447332 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.447258    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"
Sep 09 22:07:26.447332 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.447276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk" event={"ID":"61fabda6-be1f-45c1-b47c-303aa486533a","Type":"ContainerDied","Data":"a9aafad66d194861408ebff4a917d82f1b0a7443b7cd18ec688f1c0f56fd61e2"}
Sep 09 22:07:26.455303 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.455285    2585 scope.go:117] "RemoveContainer" containerID="e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f"
Sep 09 22:07:26.455618 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:07:26.455592    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f\": container with ID starting with e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f not found: ID does not exist" containerID="e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f"
Sep 09 22:07:26.455672 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.455627    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f"} err="failed to get container status \"e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f\": rpc error: code = NotFound desc = could not find container \"e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f\": container with ID starting with e39be10f2bce18bfc978c560309367b2463737a2138c3a609baa8e14914aad8f not found: ID does not exist"
Sep 09 22:07:26.455672 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.455644    2585 scope.go:117] "RemoveContainer" containerID="613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc"
Sep 09 22:07:26.463844 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.463825    2585 scope.go:117] "RemoveContainer" containerID="613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc"
Sep 09 22:07:26.464122 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:07:26.464106    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc\": container with ID starting with 613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc not found: ID does not exist" containerID="613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc"
Sep 09 22:07:26.464165 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.464129    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc"} err="failed to get container status \"613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc\": rpc error: code = NotFound desc = could not find container \"613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc\": container with ID starting with 613d574dc499eb9b22c0a94e036f44fef52bf7e3beb5da52d78c62a285479acc not found: ID does not exist"
Sep 09 22:07:26.464165 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.464145    2585 scope.go:117] "RemoveContainer" containerID="769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9"
Sep 09 22:07:26.471577 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.471557    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"]
Sep 09 22:07:26.472371 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.472355    2585 scope.go:117] "RemoveContainer" containerID="769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9"
Sep 09 22:07:26.472646 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:07:26.472626    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9\": container with ID starting with 769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9 not found: ID does not exist" containerID="769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9"
Sep 09 22:07:26.472694 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.472652    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9"} err="failed to get container status \"769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9\": rpc error: code = NotFound desc = could not find container \"769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9\": container with ID starting with 769be8d88defc7db2e8a18f21bdcb511e9b802d79ce848b48a886d49222b83e9 not found: ID does not exist"
Sep 09 22:07:26.478497 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.478477    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-100032-847c459c7c-v2t42"]
Sep 09 22:07:26.496188 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.496166    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"]
Sep 09 22:07:26.500936 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.500916    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-100032-6d8467f586-g45qb"]
Sep 09 22:07:26.515370 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.515341    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"]
Sep 09 22:07:26.519855 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:26.519835    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-100032-7f8c84c766-r5mpk"]
Sep 09 22:07:28.128278 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:28.128242    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61fabda6-be1f-45c1-b47c-303aa486533a" path="/var/lib/kubelet/pods/61fabda6-be1f-45c1-b47c-303aa486533a/volumes"
Sep 09 22:07:28.128890 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:28.128825    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72ac338-3209-4f96-bc8d-9015bfa0071b" path="/var/lib/kubelet/pods/b72ac338-3209-4f96-bc8d-9015bfa0071b/volumes"
Sep 09 22:07:28.129356 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:28.129321    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b81e0b8b-8b5c-433a-bf2a-865c5ebedab3" path="/var/lib/kubelet/pods/b81e0b8b-8b5c-433a-bf2a-865c5ebedab3/volumes"
Sep 09 22:07:31.377417 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:31.377371    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:36.124749 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:36.124717    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:36.230317 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:07:36.230266    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:08.233665 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:08.233622    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:08.549556 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:08.549452    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:14.140504 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:14.140457    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:18.185501 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:18.185469    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 22:08:18.187449 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:18.187424    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 22:08:28.893283 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:28.893239    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:28.995357 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:28.995320    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:29.076680 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:29.076648    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:29.292659 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:29.292627    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:29.915406 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:29.915370    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:30.766623 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:30.766558    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:40.280468 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.280437    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"]
Sep 09 22:08:40.280963 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.280648    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87" podUID="1cfb2665-88ba-472e-87b9-797dbebe862e" containerName="maas-api" containerID="cri-o://ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255" gracePeriod=30
Sep 09 22:08:40.300476 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.300435    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:40.323246 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.323189    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"]
Sep 09 22:08:40.323601 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.323541    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc" podUID="40919460-caca-4712-9133-5dc9144188bd" containerName="payload-processing" containerID="cri-o://5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e" gracePeriod=30
Sep 09 22:08:40.338666 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.338633    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"]
Sep 09 22:08:40.339412 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.339371    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh" podUID="81e7cd80-92e7-4865-adf5-49663deb038d" containerName="payload-pre-processing" containerID="cri-o://54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f" gracePeriod=30
Sep 09 22:08:40.375135 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.375099    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:40.445067 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.445009    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:40.516556 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.516493    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:40.610091 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.610062    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:08:40.634434 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.634412    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:08:40.640583 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.640564    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:08:40.716744 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.716707    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wg9k\" (UniqueName: \"kubernetes.io/projected/40919460-caca-4712-9133-5dc9144188bd-kube-api-access-8wg9k\") pod \"40919460-caca-4712-9133-5dc9144188bd\" (UID: \"40919460-caca-4712-9133-5dc9144188bd\") "
Sep 09 22:08:40.716940 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.716770    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfb2665-88ba-472e-87b9-797dbebe862e-openshift-service-ca\") pod \"1cfb2665-88ba-472e-87b9-797dbebe862e\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") "
Sep 09 22:08:40.716940 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.716799    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e7cd80-92e7-4865-adf5-49663deb038d-plugins-config-volume\") pod \"81e7cd80-92e7-4865-adf5-49663deb038d\" (UID: \"81e7cd80-92e7-4865-adf5-49663deb038d\") "
Sep 09 22:08:40.716940 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.716838    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w974h\" (UniqueName: \"kubernetes.io/projected/81e7cd80-92e7-4865-adf5-49663deb038d-kube-api-access-w974h\") pod \"81e7cd80-92e7-4865-adf5-49663deb038d\" (UID: \"81e7cd80-92e7-4865-adf5-49663deb038d\") "
Sep 09 22:08:40.716940 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.716881    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls\") pod \"1cfb2665-88ba-472e-87b9-797dbebe862e\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") "
Sep 09 22:08:40.717154 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.716961    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40919460-caca-4712-9133-5dc9144188bd-plugins-config-volume\") pod \"40919460-caca-4712-9133-5dc9144188bd\" (UID: \"40919460-caca-4712-9133-5dc9144188bd\") "
Sep 09 22:08:40.717154 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.716991    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-metrics-certs\") pod \"1cfb2665-88ba-472e-87b9-797dbebe862e\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") "
Sep 09 22:08:40.717154 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.717038    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75qf8\" (UniqueName: \"kubernetes.io/projected/1cfb2665-88ba-472e-87b9-797dbebe862e-kube-api-access-75qf8\") pod \"1cfb2665-88ba-472e-87b9-797dbebe862e\" (UID: \"1cfb2665-88ba-472e-87b9-797dbebe862e\") "
Sep 09 22:08:40.717420 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.717201    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfb2665-88ba-472e-87b9-797dbebe862e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1cfb2665-88ba-472e-87b9-797dbebe862e" (UID: "1cfb2665-88ba-472e-87b9-797dbebe862e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:08:40.717420 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.717259    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e7cd80-92e7-4865-adf5-49663deb038d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81e7cd80-92e7-4865-adf5-49663deb038d" (UID: "81e7cd80-92e7-4865-adf5-49663deb038d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:08:40.717547 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.717483    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40919460-caca-4712-9133-5dc9144188bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40919460-caca-4712-9133-5dc9144188bd" (UID: "40919460-caca-4712-9133-5dc9144188bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:08:40.717605 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.717559    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfb2665-88ba-472e-87b9-797dbebe862e-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:08:40.717605 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.717582    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e7cd80-92e7-4865-adf5-49663deb038d-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:08:40.717605 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.717596    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40919460-caca-4712-9133-5dc9144188bd-plugins-config-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:08:40.720080 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.720036    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e7cd80-92e7-4865-adf5-49663deb038d-kube-api-access-w974h" (OuterVolumeSpecName: "kube-api-access-w974h") pod "81e7cd80-92e7-4865-adf5-49663deb038d" (UID: "81e7cd80-92e7-4865-adf5-49663deb038d"). InnerVolumeSpecName "kube-api-access-w974h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:08:40.720433 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.720403    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40919460-caca-4712-9133-5dc9144188bd-kube-api-access-8wg9k" (OuterVolumeSpecName: "kube-api-access-8wg9k") pod "40919460-caca-4712-9133-5dc9144188bd" (UID: "40919460-caca-4712-9133-5dc9144188bd"). InnerVolumeSpecName "kube-api-access-8wg9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:08:40.720919 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.720885    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1cfb2665-88ba-472e-87b9-797dbebe862e" (UID: "1cfb2665-88ba-472e-87b9-797dbebe862e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:08:40.721042 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.720933    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1cfb2665-88ba-472e-87b9-797dbebe862e" (UID: "1cfb2665-88ba-472e-87b9-797dbebe862e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:08:40.721042 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.720983    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfb2665-88ba-472e-87b9-797dbebe862e-kube-api-access-75qf8" (OuterVolumeSpecName: "kube-api-access-75qf8") pod "1cfb2665-88ba-472e-87b9-797dbebe862e" (UID: "1cfb2665-88ba-472e-87b9-797dbebe862e"). InnerVolumeSpecName "kube-api-access-75qf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:08:40.813500 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.813397    2585 generic.go:356] "Generic (PLEG): container finished" podID="1cfb2665-88ba-472e-87b9-797dbebe862e" containerID="ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255" exitCode=0
Sep 09 22:08:40.813500 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.813442    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"
Sep 09 22:08:40.813500 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.813461    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87" event={"ID":"1cfb2665-88ba-472e-87b9-797dbebe862e","Type":"ContainerDied","Data":"ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255"}
Sep 09 22:08:40.813778 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.813516    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87" event={"ID":"1cfb2665-88ba-472e-87b9-797dbebe862e","Type":"ContainerDied","Data":"158d567fb6e696ec5bb867f1ce0c1b1d913172bc428b7edbb0d75ea4c126d7c4"}
Sep 09 22:08:40.813778 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.813550    2585 scope.go:117] "RemoveContainer" containerID="ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255"
Sep 09 22:08:40.814915 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.814893    2585 generic.go:356] "Generic (PLEG): container finished" podID="81e7cd80-92e7-4865-adf5-49663deb038d" containerID="54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f" exitCode=0
Sep 09 22:08:40.815018 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.814935    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"
Sep 09 22:08:40.815018 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.814977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh" event={"ID":"81e7cd80-92e7-4865-adf5-49663deb038d","Type":"ContainerDied","Data":"54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f"}
Sep 09 22:08:40.815134 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.815025    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh" event={"ID":"81e7cd80-92e7-4865-adf5-49663deb038d","Type":"ContainerDied","Data":"5007c5d8b42e25ed29da15e62bf022d37b4edf38e800f6b2a3e14769bde8db84"}
Sep 09 22:08:40.816527 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.816493    2585 generic.go:356] "Generic (PLEG): container finished" podID="40919460-caca-4712-9133-5dc9144188bd" containerID="5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e" exitCode=0
Sep 09 22:08:40.816689 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.816544    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc" event={"ID":"40919460-caca-4712-9133-5dc9144188bd","Type":"ContainerDied","Data":"5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e"}
Sep 09 22:08:40.816689 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.816552    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"
Sep 09 22:08:40.816689 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.816590    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc" event={"ID":"40919460-caca-4712-9133-5dc9144188bd","Type":"ContainerDied","Data":"4e57815a582c4f383dd91178f8fdb3fa9fc5f91effd6cbbe2f4e80124aacf6e7"}
Sep 09 22:08:40.818196 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.818176    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qf8\" (UniqueName: \"kubernetes.io/projected/1cfb2665-88ba-472e-87b9-797dbebe862e-kube-api-access-75qf8\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:08:40.818196 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.818195    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wg9k\" (UniqueName: \"kubernetes.io/projected/40919460-caca-4712-9133-5dc9144188bd-kube-api-access-8wg9k\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:08:40.818196 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.818221    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w974h\" (UniqueName: \"kubernetes.io/projected/81e7cd80-92e7-4865-adf5-49663deb038d-kube-api-access-w974h\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:08:40.818196 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.818243    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:08:40.818533 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.818253    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfb2665-88ba-472e-87b9-797dbebe862e-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:08:40.833417 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.833118    2585 scope.go:117] "RemoveContainer" containerID="ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255"
Sep 09 22:08:40.835154 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:08:40.835124    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255\": container with ID starting with ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255 not found: ID does not exist" containerID="ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255"
Sep 09 22:08:40.835295 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.835162    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255"} err="failed to get container status \"ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255\": rpc error: code = NotFound desc = could not find container \"ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255\": container with ID starting with ba5d280883949c523b3ae23e70470238a357a8e8aaceed7046754be776df1255 not found: ID does not exist"
Sep 09 22:08:40.835534 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.835402    2585 scope.go:117] "RemoveContainer" containerID="54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f"
Sep 09 22:08:40.844162 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.844140    2585 scope.go:117] "RemoveContainer" containerID="54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f"
Sep 09 22:08:40.844513 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:08:40.844485    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f\": container with ID starting with 54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f not found: ID does not exist" containerID="54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f"
Sep 09 22:08:40.844621 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.844525    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f"} err="failed to get container status \"54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f\": rpc error: code = NotFound desc = could not find container \"54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f\": container with ID starting with 54c02c6f8008ac7ae21cb8cc998bbc212629e602a917e3e8b9f30a768bb06e1f not found: ID does not exist"
Sep 09 22:08:40.844621 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.844542    2585 scope.go:117] "RemoveContainer" containerID="5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e"
Sep 09 22:08:40.846883 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.846861    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"]
Sep 09 22:08:40.851583 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.851561    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-855e86-7f7dcd6d4b-77r87"]
Sep 09 22:08:40.852819 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.852803    2585 scope.go:117] "RemoveContainer" containerID="5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e"
Sep 09 22:08:40.853079 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:08:40.853063    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e\": container with ID starting with 5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e not found: ID does not exist" containerID="5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e"
Sep 09 22:08:40.853141 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.853084    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e"} err="failed to get container status \"5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e\": rpc error: code = NotFound desc = could not find container \"5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e\": container with ID starting with 5f1e7d596ac4f8ac27fe5d5e685c011402b0c39f2ab22b6e04eccd9ffbeb137e not found: ID does not exist"
Sep 09 22:08:40.863234 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.863196    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"]
Sep 09 22:08:40.869833 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.869813    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-855e86-5879d78cd-hcqzh"]
Sep 09 22:08:40.880541 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.880522    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"]
Sep 09 22:08:40.885771 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.885750    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-855e86-546d9bc677-s6lxc"]
Sep 09 22:08:40.925254 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:40.925188    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:42.128030 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:42.127997    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cfb2665-88ba-472e-87b9-797dbebe862e" path="/var/lib/kubelet/pods/1cfb2665-88ba-472e-87b9-797dbebe862e/volumes"
Sep 09 22:08:42.128428 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:42.128375    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40919460-caca-4712-9133-5dc9144188bd" path="/var/lib/kubelet/pods/40919460-caca-4712-9133-5dc9144188bd/volumes"
Sep 09 22:08:42.128678 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:42.128665    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e7cd80-92e7-4865-adf5-49663deb038d" path="/var/lib/kubelet/pods/81e7cd80-92e7-4865-adf5-49663deb038d/volumes"
Sep 09 22:08:47.193061 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:47.193023    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:47.292165 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:47.292128    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:47.501456 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:47.501366    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:48.000202 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:48.000168    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:58.380636 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:58.380600    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:58.451137 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:58.451104    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:58.529312 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:58.529257    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:58.628161 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:58.628127    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:58.993678 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:08:58.993644    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:03.707322 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:03.707286    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:05.281184 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:05.281152    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:05.382583 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:05.382549    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:05.709983 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:05.709946    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:06.301909 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:06.301879    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:16.585087 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.584973    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"]
Sep 09 22:09:16.585587 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.585253    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g" podUID="c3087450-ec4c-47e5-bfcc-09b6424cb64e" containerName="maas-api" containerID="cri-o://ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54" gracePeriod=30
Sep 09 22:09:16.629931 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.629900    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:16.734835 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.734799    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:16.831569 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.831542    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 22:09:16.834361 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.834330    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:16.934072 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.934036    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:16.941914 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.941800    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-maas-api-tls\") pod \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") "
Sep 09 22:09:16.941914 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.941867    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-metrics-certs\") pod \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") "
Sep 09 22:09:16.942126 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.941939    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4stqt\" (UniqueName: \"kubernetes.io/projected/c3087450-ec4c-47e5-bfcc-09b6424cb64e-kube-api-access-4stqt\") pod \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") "
Sep 09 22:09:16.942126 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.941989    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3087450-ec4c-47e5-bfcc-09b6424cb64e-openshift-service-ca\") pod \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\" (UID: \"c3087450-ec4c-47e5-bfcc-09b6424cb64e\") "
Sep 09 22:09:16.942853 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.942816    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3087450-ec4c-47e5-bfcc-09b6424cb64e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3087450-ec4c-47e5-bfcc-09b6424cb64e" (UID: "c3087450-ec4c-47e5-bfcc-09b6424cb64e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:16.945849 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.945814    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3087450-ec4c-47e5-bfcc-09b6424cb64e" (UID: "c3087450-ec4c-47e5-bfcc-09b6424cb64e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:09:16.948867 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.948043    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3087450-ec4c-47e5-bfcc-09b6424cb64e-kube-api-access-4stqt" (OuterVolumeSpecName: "kube-api-access-4stqt") pod "c3087450-ec4c-47e5-bfcc-09b6424cb64e" (UID: "c3087450-ec4c-47e5-bfcc-09b6424cb64e"). InnerVolumeSpecName "kube-api-access-4stqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:16.948985 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.948951    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c3087450-ec4c-47e5-bfcc-09b6424cb64e" (UID: "c3087450-ec4c-47e5-bfcc-09b6424cb64e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:09:16.990351 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.990311    2585 generic.go:356] "Generic (PLEG): container finished" podID="c3087450-ec4c-47e5-bfcc-09b6424cb64e" containerID="ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54" exitCode=0
Sep 09 22:09:16.990570 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.990394    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"
Sep 09 22:09:16.990570 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.990391    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g" event={"ID":"c3087450-ec4c-47e5-bfcc-09b6424cb64e","Type":"ContainerDied","Data":"ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54"}
Sep 09 22:09:16.990570 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.990509    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g" event={"ID":"c3087450-ec4c-47e5-bfcc-09b6424cb64e","Type":"ContainerDied","Data":"3d409a2b7394a388b4b9745e5ef97492a55391e23659c8aa31bad25082a5ca30"}
Sep 09 22:09:16.990570 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.990543    2585 scope.go:117] "RemoveContainer" containerID="ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54"
Sep 09 22:09:16.999975 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:16.999956    2585 scope.go:117] "RemoveContainer" containerID="ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54"
Sep 09 22:09:17.000283 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:09:17.000262    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54\": container with ID starting with ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54 not found: ID does not exist" containerID="ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54"
Sep 09 22:09:17.000360 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:17.000294    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54"} err="failed to get container status \"ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54\": rpc error: code = NotFound desc = could not find container \"ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54\": container with ID starting with ab5a05da30b80974ffa07896e4eef3500f823fd9ea3a9cd3ebe33b88db51ea54 not found: ID does not exist"
Sep 09 22:09:17.013864 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:17.013842    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"]
Sep 09 22:09:17.021877 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:17.021855    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-64fec6-b64f4bd66-ftl9g"]
Sep 09 22:09:17.043592 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:17.043569    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4stqt\" (UniqueName: \"kubernetes.io/projected/c3087450-ec4c-47e5-bfcc-09b6424cb64e-kube-api-access-4stqt\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:09:17.043592 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:17.043590    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3087450-ec4c-47e5-bfcc-09b6424cb64e-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:09:17.043719 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:17.043600    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:09:17.043719 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:17.043610    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3087450-ec4c-47e5-bfcc-09b6424cb64e-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:09:17.279567 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:17.279474    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:18.128836 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:18.128801    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3087450-ec4c-47e5-bfcc-09b6424cb64e" path="/var/lib/kubelet/pods/c3087450-ec4c-47e5-bfcc-09b6424cb64e/volumes"
Sep 09 22:09:23.333774 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:23.333737    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:23.391778 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:23.391743    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:23.608143 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:23.608055    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:26.471999 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.471961    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4f964485-rr8r5"]
Sep 09 22:09:26.472441 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.472162    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c4f964485-rr8r5" podUID="480d0adf-08ef-4d0c-a495-c9d1ecb54532" containerName="manager" containerID="cri-o://05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81" gracePeriod=30
Sep 09 22:09:26.725615 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.725554    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 22:09:26.835662 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.835626    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsjzn\" (UniqueName: \"kubernetes.io/projected/480d0adf-08ef-4d0c-a495-c9d1ecb54532-kube-api-access-rsjzn\") pod \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") "
Sep 09 22:09:26.835662 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.835664    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-webhook-cert\") pod \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") "
Sep 09 22:09:26.835889 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.835729    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-metrics-certs\") pod \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\" (UID: \"480d0adf-08ef-4d0c-a495-c9d1ecb54532\") "
Sep 09 22:09:26.837996 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.837957    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "480d0adf-08ef-4d0c-a495-c9d1ecb54532" (UID: "480d0adf-08ef-4d0c-a495-c9d1ecb54532"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:09:26.838111 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.838011    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "480d0adf-08ef-4d0c-a495-c9d1ecb54532" (UID: "480d0adf-08ef-4d0c-a495-c9d1ecb54532"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:09:26.838111 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.838012    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480d0adf-08ef-4d0c-a495-c9d1ecb54532-kube-api-access-rsjzn" (OuterVolumeSpecName: "kube-api-access-rsjzn") pod "480d0adf-08ef-4d0c-a495-c9d1ecb54532" (UID: "480d0adf-08ef-4d0c-a495-c9d1ecb54532"). InnerVolumeSpecName "kube-api-access-rsjzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:26.936592 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.936552    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:09:26.936592 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.936587    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsjzn\" (UniqueName: \"kubernetes.io/projected/480d0adf-08ef-4d0c-a495-c9d1ecb54532-kube-api-access-rsjzn\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:09:26.936592 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:26.936601    2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480d0adf-08ef-4d0c-a495-c9d1ecb54532-webhook-cert\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:09:27.043174 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.043080    2585 generic.go:356] "Generic (PLEG): container finished" podID="480d0adf-08ef-4d0c-a495-c9d1ecb54532" containerID="05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81" exitCode=0
Sep 09 22:09:27.043174 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.043127    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-rr8r5"
Sep 09 22:09:27.043174 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.043147    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-rr8r5" event={"ID":"480d0adf-08ef-4d0c-a495-c9d1ecb54532","Type":"ContainerDied","Data":"05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81"}
Sep 09 22:09:27.043461 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.043191    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-rr8r5" event={"ID":"480d0adf-08ef-4d0c-a495-c9d1ecb54532","Type":"ContainerDied","Data":"564f70c979e426ee4941873c919f222ca27dd09da41a6f877eefb0fbd3385782"}
Sep 09 22:09:27.043461 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.043235    2585 scope.go:117] "RemoveContainer" containerID="05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81"
Sep 09 22:09:27.052838 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.052815    2585 scope.go:117] "RemoveContainer" containerID="05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81"
Sep 09 22:09:27.053104 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:09:27.053086    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81\": container with ID starting with 05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81 not found: ID does not exist" containerID="05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81"
Sep 09 22:09:27.053175 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.053111    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81"} err="failed to get container status \"05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81\": rpc error: code = NotFound desc = could not find container \"05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81\": container with ID starting with 05da31c2bf33f2fd8a83f3a2327e055dd33b4ff2e2fe8aa5c26be01ac1e9ff81 not found: ID does not exist"
Sep 09 22:09:27.065392 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.065370    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4f964485-rr8r5"]
Sep 09 22:09:27.069764 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:27.069744    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c4f964485-rr8r5"]
Sep 09 22:09:28.128511 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:09:28.128475    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480d0adf-08ef-4d0c-a495-c9d1ecb54532" path="/var/lib/kubelet/pods/480d0adf-08ef-4d0c-a495-c9d1ecb54532/volumes"
Sep 09 22:10:27.959948 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.959915    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4f964485-xszfd"]
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960277    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40919460-caca-4712-9133-5dc9144188bd" containerName="payload-processing"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960289    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="40919460-caca-4712-9133-5dc9144188bd" containerName="payload-processing"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960304    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e7cd80-92e7-4865-adf5-49663deb038d" containerName="payload-pre-processing"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960310    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e7cd80-92e7-4865-adf5-49663deb038d" containerName="payload-pre-processing"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960319    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480d0adf-08ef-4d0c-a495-c9d1ecb54532" containerName="manager"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960324    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="480d0adf-08ef-4d0c-a495-c9d1ecb54532" containerName="manager"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960333    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3087450-ec4c-47e5-bfcc-09b6424cb64e" containerName="maas-api"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960338    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3087450-ec4c-47e5-bfcc-09b6424cb64e" containerName="maas-api"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960349    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72ac338-3209-4f96-bc8d-9015bfa0071b" containerName="payload-processing"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960354    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72ac338-3209-4f96-bc8d-9015bfa0071b" containerName="payload-processing"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960362    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cfb2665-88ba-472e-87b9-797dbebe862e" containerName="maas-api"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960368    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cfb2665-88ba-472e-87b9-797dbebe862e" containerName="maas-api"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960375    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fabda6-be1f-45c1-b47c-303aa486533a" containerName="maas-api"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960380    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fabda6-be1f-45c1-b47c-303aa486533a" containerName="maas-api"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960387    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b81e0b8b-8b5c-433a-bf2a-865c5ebedab3" containerName="payload-pre-processing"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960392    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b81e0b8b-8b5c-433a-bf2a-865c5ebedab3" containerName="payload-pre-processing"
Sep 09 22:10:27.960438 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960444    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e7cd80-92e7-4865-adf5-49663deb038d" containerName="payload-pre-processing"
Sep 09 22:10:27.961185 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960452    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="61fabda6-be1f-45c1-b47c-303aa486533a" containerName="maas-api"
Sep 09 22:10:27.961185 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960459    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="480d0adf-08ef-4d0c-a495-c9d1ecb54532" containerName="manager"
Sep 09 22:10:27.961185 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960464    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="40919460-caca-4712-9133-5dc9144188bd" containerName="payload-processing"
Sep 09 22:10:27.961185 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960469    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cfb2665-88ba-472e-87b9-797dbebe862e" containerName="maas-api"
Sep 09 22:10:27.961185 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960476    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b81e0b8b-8b5c-433a-bf2a-865c5ebedab3" containerName="payload-pre-processing"
Sep 09 22:10:27.961185 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960484    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72ac338-3209-4f96-bc8d-9015bfa0071b" containerName="payload-processing"
Sep 09 22:10:27.961185 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.960490    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3087450-ec4c-47e5-bfcc-09b6424cb64e" containerName="maas-api"
Sep 09 22:10:27.963522 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.963504    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:27.966621 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.966592    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dqzb\""
Sep 09 22:10:27.966621 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.966595    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 22:10:27.966827 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.966600    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 22:10:27.966827 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.966633    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 22:10:27.973811 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.973788    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4f964485-xszfd"]
Sep 09 22:10:27.976616 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.976593    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-metrics-certs\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:27.976728 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.976653    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnm9z\" (UniqueName: \"kubernetes.io/projected/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-kube-api-access-pnm9z\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:27.976728 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:27.976718    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-webhook-cert\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:28.077886 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.077846    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-metrics-certs\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:28.078087 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.077946    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnm9z\" (UniqueName: \"kubernetes.io/projected/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-kube-api-access-pnm9z\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:28.078087 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.078004    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-webhook-cert\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:28.080664 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.080631    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-metrics-certs\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:28.080785 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.080679    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-webhook-cert\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:28.087223 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.087188    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnm9z\" (UniqueName: \"kubernetes.io/projected/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-kube-api-access-pnm9z\") pod \"maas-controller-c4f964485-xszfd\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") " pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:28.275130 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.275030    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:28.547153 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.547129    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4f964485-xszfd"]
Sep 09 22:10:28.549626 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:10:28.549600    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c1599b_948d_4cbb_b3e0_d16fed9548fb.slice/crio-2b35469898e76a70da6f310bc9df31f630eee8430ed206a1705614e90434f693 WatchSource:0}: Error finding container 2b35469898e76a70da6f310bc9df31f630eee8430ed206a1705614e90434f693: Status 404 returned error can't find the container with id 2b35469898e76a70da6f310bc9df31f630eee8430ed206a1705614e90434f693
Sep 09 22:10:28.551283 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:28.551268    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 22:10:29.329189 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:29.329151    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-xszfd" event={"ID":"e9c1599b-948d-4cbb-b3e0-d16fed9548fb","Type":"ContainerStarted","Data":"b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d"}
Sep 09 22:10:29.329585 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:29.329239    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-xszfd" event={"ID":"e9c1599b-948d-4cbb-b3e0-d16fed9548fb","Type":"ContainerStarted","Data":"2b35469898e76a70da6f310bc9df31f630eee8430ed206a1705614e90434f693"}
Sep 09 22:10:29.329585 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:29.329269    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:29.346556 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:29.346504    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c4f964485-xszfd" podStartSLOduration=1.756458367 podStartE2EDuration="2.346487543s" totalImagesPullingTime="590.029176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:10:29.344232424 +0000 UTC m=+2231.718556245" watchObservedRunningTime="2026-09-09 22:10:29.346487543 +0000 UTC m=+2231.720811742"
Sep 09 22:10:40.336879 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:40.336797    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:47.989162 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:47.989129    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:49.709458 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:49.709411    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:49.809322 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:49.809286    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4f964485-xszfd"]
Sep 09 22:10:49.809587 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:49.809547    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c4f964485-xszfd" podUID="e9c1599b-948d-4cbb-b3e0-d16fed9548fb" containerName="manager" containerID="cri-o://b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d" gracePeriod=30
Sep 09 22:10:50.046805 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.046778    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:50.184502 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.184473    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-metrics-certs\") pod \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") "
Sep 09 22:10:50.184685 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.184524    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnm9z\" (UniqueName: \"kubernetes.io/projected/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-kube-api-access-pnm9z\") pod \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") "
Sep 09 22:10:50.184685 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.184545    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-webhook-cert\") pod \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\" (UID: \"e9c1599b-948d-4cbb-b3e0-d16fed9548fb\") "
Sep 09 22:10:50.186869 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.186838    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9c1599b-948d-4cbb-b3e0-d16fed9548fb" (UID: "e9c1599b-948d-4cbb-b3e0-d16fed9548fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:10:50.186985 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.186866    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-kube-api-access-pnm9z" (OuterVolumeSpecName: "kube-api-access-pnm9z") pod "e9c1599b-948d-4cbb-b3e0-d16fed9548fb" (UID: "e9c1599b-948d-4cbb-b3e0-d16fed9548fb"). InnerVolumeSpecName "kube-api-access-pnm9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:10:50.186985 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.186871    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e9c1599b-948d-4cbb-b3e0-d16fed9548fb" (UID: "e9c1599b-948d-4cbb-b3e0-d16fed9548fb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:10:50.285895 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.285808    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:10:50.285895 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.285838    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnm9z\" (UniqueName: \"kubernetes.io/projected/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-kube-api-access-pnm9z\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:10:50.285895 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.285847    2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9c1599b-948d-4cbb-b3e0-d16fed9548fb-webhook-cert\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:10:50.434035 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.433998    2585 generic.go:356] "Generic (PLEG): container finished" podID="e9c1599b-948d-4cbb-b3e0-d16fed9548fb" containerID="b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d" exitCode=0
Sep 09 22:10:50.434223 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.434039    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-xszfd" event={"ID":"e9c1599b-948d-4cbb-b3e0-d16fed9548fb","Type":"ContainerDied","Data":"b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d"}
Sep 09 22:10:50.434223 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.434058    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-xszfd"
Sep 09 22:10:50.434223 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.434109    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-xszfd" event={"ID":"e9c1599b-948d-4cbb-b3e0-d16fed9548fb","Type":"ContainerDied","Data":"2b35469898e76a70da6f310bc9df31f630eee8430ed206a1705614e90434f693"}
Sep 09 22:10:50.434223 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.434137    2585 scope.go:117] "RemoveContainer" containerID="b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d"
Sep 09 22:10:50.444885 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.444864    2585 scope.go:117] "RemoveContainer" containerID="b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d"
Sep 09 22:10:50.445181 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:10:50.445159    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d\": container with ID starting with b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d not found: ID does not exist" containerID="b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d"
Sep 09 22:10:50.445279 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.445194    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d"} err="failed to get container status \"b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d\": rpc error: code = NotFound desc = could not find container \"b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d\": container with ID starting with b1327f5bee5086f584ff6a095a4ada698d409767861504a133f9f2f71704147d not found: ID does not exist"
Sep 09 22:10:50.454981 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.454955    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4f964485-xszfd"]
Sep 09 22:10:50.460732 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:50.460709    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c4f964485-xszfd"]
Sep 09 22:10:51.176234 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.176171    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:51.290996 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.290953    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4f964485-z6vhn"]
Sep 09 22:10:51.291398 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.291380    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c1599b-948d-4cbb-b3e0-d16fed9548fb" containerName="manager"
Sep 09 22:10:51.291502 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.291402    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c1599b-948d-4cbb-b3e0-d16fed9548fb" containerName="manager"
Sep 09 22:10:51.291557 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.291531    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9c1599b-948d-4cbb-b3e0-d16fed9548fb" containerName="manager"
Sep 09 22:10:51.295762 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.295723    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.298711 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.298689    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 22:10:51.298840 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.298727    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 22:10:51.298840 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.298753    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 22:10:51.298840 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.298727    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dqzb\""
Sep 09 22:10:51.303562 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.303447    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4f964485-z6vhn"]
Sep 09 22:10:51.396626 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.396587    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-webhook-cert\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.396816 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.396647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-metrics-certs\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.396816 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.396690    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47x9r\" (UniqueName: \"kubernetes.io/projected/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-kube-api-access-47x9r\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.497770 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.497689    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-webhook-cert\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.497770 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.497739    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-metrics-certs\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.497955 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.497780    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47x9r\" (UniqueName: \"kubernetes.io/projected/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-kube-api-access-47x9r\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.500571 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.500542    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-metrics-certs\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.500701 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.500542    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-webhook-cert\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.505611 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.505591    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47x9r\" (UniqueName: \"kubernetes.io/projected/fe266c2f-60ed-4cfb-86a4-d9b202c0232d-kube-api-access-47x9r\") pod \"maas-controller-c4f964485-z6vhn\" (UID: \"fe266c2f-60ed-4cfb-86a4-d9b202c0232d\") " pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.608178 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.608124    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:51.880393 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:51.880368    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4f964485-z6vhn"]
Sep 09 22:10:51.882888 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:10:51.882847    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe266c2f_60ed_4cfb_86a4_d9b202c0232d.slice/crio-6ea31d2d107e12bfcf0ea2d11e997c0d90de46f50fd36d8a3b5cfba83664f22f WatchSource:0}: Error finding container 6ea31d2d107e12bfcf0ea2d11e997c0d90de46f50fd36d8a3b5cfba83664f22f: Status 404 returned error can't find the container with id 6ea31d2d107e12bfcf0ea2d11e997c0d90de46f50fd36d8a3b5cfba83664f22f
Sep 09 22:10:52.127139 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:52.127104    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9c1599b-948d-4cbb-b3e0-d16fed9548fb" path="/var/lib/kubelet/pods/e9c1599b-948d-4cbb-b3e0-d16fed9548fb/volumes"
Sep 09 22:10:52.445731 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:52.445696    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-z6vhn" event={"ID":"fe266c2f-60ed-4cfb-86a4-d9b202c0232d","Type":"ContainerStarted","Data":"2f52c0e6c1107dd89d0a9fd365fbb147999f19544e8f63f8ee6c64b3e68f5e64"}
Sep 09 22:10:52.446122 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:52.445745    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4f964485-z6vhn" event={"ID":"fe266c2f-60ed-4cfb-86a4-d9b202c0232d","Type":"ContainerStarted","Data":"6ea31d2d107e12bfcf0ea2d11e997c0d90de46f50fd36d8a3b5cfba83664f22f"}
Sep 09 22:10:52.446122 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:52.445789    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:10:52.464091 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:10:52.463987    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c4f964485-z6vhn" podStartSLOduration=1.142163662 podStartE2EDuration="1.46397194s" totalImagesPullingTime="321.808278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:10:52.462902648 +0000 UTC m=+2254.837226476" watchObservedRunningTime="2026-09-09 22:10:52.46397194 +0000 UTC m=+2254.838295764"
Sep 09 22:11:03.453346 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:11:03.453316    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4f964485-z6vhn"
Sep 09 22:11:09.662667 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:11:09.662629    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:48.426493 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:11:48.426459    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:59.597510 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:11:59.597465    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"]
Sep 09 22:11:59.597993 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:11:59.597669    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k" podUID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerName="manager" containerID="cri-o://125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f" gracePeriod=10
Sep 09 22:12:00.550987 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.550959    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 22:12:00.627371 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.627287    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90e25932-0d31-4c53-944a-4a3dd2093b20-extensions-socket-volume\") pod \"90e25932-0d31-4c53-944a-4a3dd2093b20\" (UID: \"90e25932-0d31-4c53-944a-4a3dd2093b20\") "
Sep 09 22:12:00.627371 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.627340    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjvw4\" (UniqueName: \"kubernetes.io/projected/90e25932-0d31-4c53-944a-4a3dd2093b20-kube-api-access-vjvw4\") pod \"90e25932-0d31-4c53-944a-4a3dd2093b20\" (UID: \"90e25932-0d31-4c53-944a-4a3dd2093b20\") "
Sep 09 22:12:00.627775 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.627742    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e25932-0d31-4c53-944a-4a3dd2093b20-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "90e25932-0d31-4c53-944a-4a3dd2093b20" (UID: "90e25932-0d31-4c53-944a-4a3dd2093b20"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:12:00.629664 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.629641    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e25932-0d31-4c53-944a-4a3dd2093b20-kube-api-access-vjvw4" (OuterVolumeSpecName: "kube-api-access-vjvw4") pod "90e25932-0d31-4c53-944a-4a3dd2093b20" (UID: "90e25932-0d31-4c53-944a-4a3dd2093b20"). InnerVolumeSpecName "kube-api-access-vjvw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:12:00.728367 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.728322    2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90e25932-0d31-4c53-944a-4a3dd2093b20-extensions-socket-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:12:00.728367 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.728356    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvw4\" (UniqueName: \"kubernetes.io/projected/90e25932-0d31-4c53-944a-4a3dd2093b20-kube-api-access-vjvw4\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:12:00.764471 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.764440    2585 generic.go:356] "Generic (PLEG): container finished" podID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerID="125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f" exitCode=0
Sep 09 22:12:00.764624 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.764492    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"
Sep 09 22:12:00.764624 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.764520    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k" event={"ID":"90e25932-0d31-4c53-944a-4a3dd2093b20","Type":"ContainerDied","Data":"125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f"}
Sep 09 22:12:00.764624 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.764574    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k" event={"ID":"90e25932-0d31-4c53-944a-4a3dd2093b20","Type":"ContainerDied","Data":"4a6e652a499781be1d9f89883de77d9a8b0e3abf00ff54a628634d8fe49b6140"}
Sep 09 22:12:00.764624 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.764599    2585 scope.go:117] "RemoveContainer" containerID="125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f"
Sep 09 22:12:00.773917 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.773900    2585 scope.go:117] "RemoveContainer" containerID="50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5"
Sep 09 22:12:00.781970 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.781954    2585 scope.go:117] "RemoveContainer" containerID="125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f"
Sep 09 22:12:00.782272 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:12:00.782251    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f\": container with ID starting with 125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f not found: ID does not exist" containerID="125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f"
Sep 09 22:12:00.782316 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.782282    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f"} err="failed to get container status \"125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f\": rpc error: code = NotFound desc = could not find container \"125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f\": container with ID starting with 125485759e518d7716e6c3e9c19589099856c6959c9e20c4b20742d3b933d28f not found: ID does not exist"
Sep 09 22:12:00.782316 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.782299    2585 scope.go:117] "RemoveContainer" containerID="50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5"
Sep 09 22:12:00.782566 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:12:00.782545    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5\": container with ID starting with 50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5 not found: ID does not exist" containerID="50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5"
Sep 09 22:12:00.782643 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.782577    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5"} err="failed to get container status \"50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5\": rpc error: code = NotFound desc = could not find container \"50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5\": container with ID starting with 50116144e523353f1f8707fb950dade68c01e069f0faf481c440ac6c7bd0eda5 not found: ID does not exist"
Sep 09 22:12:00.785377 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.785353    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"]
Sep 09 22:12:00.788460 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:00.788439    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8mp2k"]
Sep 09 22:12:02.127769 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:12:02.127728    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90e25932-0d31-4c53-944a-4a3dd2093b20" path="/var/lib/kubelet/pods/90e25932-0d31-4c53-944a-4a3dd2093b20/volumes"
Sep 09 22:13:00.396474 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.396435    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"]
Sep 09 22:13:00.396934 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.396793    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerName="manager"
Sep 09 22:13:00.396934 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.396806    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerName="manager"
Sep 09 22:13:00.396934 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.396825    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerName="manager"
Sep 09 22:13:00.396934 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.396830    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerName="manager"
Sep 09 22:13:00.396934 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.396895    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerName="manager"
Sep 09 22:13:00.396934 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.396906    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="90e25932-0d31-4c53-944a-4a3dd2093b20" containerName="manager"
Sep 09 22:13:00.399642 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.399625    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:00.402368 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.402348    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rx9gw\""
Sep 09 22:13:00.416171 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.416143    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"]
Sep 09 22:13:00.458530 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.458499    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19341b3e-e27a-432f-8d22-22b4c344ff9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hmktt\" (UID: \"19341b3e-e27a-432f-8d22-22b4c344ff9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:00.458530 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.458533    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvsx\" (UniqueName: \"kubernetes.io/projected/19341b3e-e27a-432f-8d22-22b4c344ff9c-kube-api-access-vfvsx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hmktt\" (UID: \"19341b3e-e27a-432f-8d22-22b4c344ff9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:00.559836 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.559796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19341b3e-e27a-432f-8d22-22b4c344ff9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hmktt\" (UID: \"19341b3e-e27a-432f-8d22-22b4c344ff9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:00.559836 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.559839    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvsx\" (UniqueName: \"kubernetes.io/projected/19341b3e-e27a-432f-8d22-22b4c344ff9c-kube-api-access-vfvsx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hmktt\" (UID: \"19341b3e-e27a-432f-8d22-22b4c344ff9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:00.560275 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.560253    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19341b3e-e27a-432f-8d22-22b4c344ff9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hmktt\" (UID: \"19341b3e-e27a-432f-8d22-22b4c344ff9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:00.570503 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.570474    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvsx\" (UniqueName: \"kubernetes.io/projected/19341b3e-e27a-432f-8d22-22b4c344ff9c-kube-api-access-vfvsx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hmktt\" (UID: \"19341b3e-e27a-432f-8d22-22b4c344ff9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:00.709539 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.709492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:00.990267 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:00.990238    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"]
Sep 09 22:13:00.992334 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:13:00.992306    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19341b3e_e27a_432f_8d22_22b4c344ff9c.slice/crio-5e131d2dd86a026b3a8b4b16a636c17e5106b4846e1e726ca11b3f8cec174ff4 WatchSource:0}: Error finding container 5e131d2dd86a026b3a8b4b16a636c17e5106b4846e1e726ca11b3f8cec174ff4: Status 404 returned error can't find the container with id 5e131d2dd86a026b3a8b4b16a636c17e5106b4846e1e726ca11b3f8cec174ff4
Sep 09 22:13:01.038617 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:01.038590    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt" event={"ID":"19341b3e-e27a-432f-8d22-22b4c344ff9c","Type":"ContainerStarted","Data":"5e131d2dd86a026b3a8b4b16a636c17e5106b4846e1e726ca11b3f8cec174ff4"}
Sep 09 22:13:02.050930 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:02.050888    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt" event={"ID":"19341b3e-e27a-432f-8d22-22b4c344ff9c","Type":"ContainerStarted","Data":"3324d30d981d6ea29cd280b53cb326170e4552219b47f5ae4bef9adae9f1fe2e"}
Sep 09 22:13:02.051451 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:02.050988    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:02.074020 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:02.073965    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt" podStartSLOduration=2.073949587 podStartE2EDuration="2.073949587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:13:02.072881917 +0000 UTC m=+2384.447205742" watchObservedRunningTime="2026-09-09 22:13:02.073949587 +0000 UTC m=+2384.448273408"
Sep 09 22:13:13.059144 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:13.059110    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hmktt"
Sep 09 22:13:18.216291 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:18.216254    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 22:13:18.218913 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:18.218881    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w5txw_257e50ad-c41f-420c-b697-a1ea0d80bb2d/ovn-acl-logging/0.log"
Sep 09 22:13:37.837046 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:37.837002    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:13:42.485835 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.485755    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"]
Sep 09 22:13:42.489557 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.489531    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.492308 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.492285    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fcc30655-kserve-self-signed-certs\""
Sep 09 22:13:42.499668 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.499639    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"]
Sep 09 22:13:42.517412 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.517378    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:13:42.544586 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.544552    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-model-cache\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.544743 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.544639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tmp-dir\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.544743 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.544681    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hctv\" (UniqueName: \"kubernetes.io/projected/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-kube-api-access-2hctv\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.544743 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.544714    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-home\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.544879 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.544764    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tls-certs\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.544879 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.544805    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-dshm\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.645847 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.645793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tls-certs\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.645847 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.645849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-dshm\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.646100 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.645891    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-model-cache\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.646100 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.645949    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tmp-dir\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.646100 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.645994    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hctv\" (UniqueName: \"kubernetes.io/projected/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-kube-api-access-2hctv\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.646100 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.646034    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-home\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.646520 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.646467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-model-cache\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.646520 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.646503    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tmp-dir\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.646656 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.646553    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-home\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.648476 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.648456    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-dshm\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.648917 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.648896    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tls-certs\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.654546 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.654526    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hctv\" (UniqueName: \"kubernetes.io/projected/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-kube-api-access-2hctv\") pod \"e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") " pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:42.801015 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:42.800899    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:43.078072 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:43.078007    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"]
Sep 09 22:13:43.080357 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:13:43.080321    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9c5cd64_e565_47a9_b55f_5f6a9c2f8bb0.slice/crio-1f5636d66c8a25a5fbec0236e6d0d1842ce8b4c6b0fc227073025a4c9c629e65 WatchSource:0}: Error finding container 1f5636d66c8a25a5fbec0236e6d0d1842ce8b4c6b0fc227073025a4c9c629e65: Status 404 returned error can't find the container with id 1f5636d66c8a25a5fbec0236e6d0d1842ce8b4c6b0fc227073025a4c9c629e65
Sep 09 22:13:43.247840 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:43.247803    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml" event={"ID":"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0","Type":"ContainerStarted","Data":"db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e"}
Sep 09 22:13:43.248022 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:43.247849    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml" event={"ID":"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0","Type":"ContainerStarted","Data":"1f5636d66c8a25a5fbec0236e6d0d1842ce8b4c6b0fc227073025a4c9c629e65"}
Sep 09 22:13:43.264452 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:43.264404    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml" podStartSLOduration=1.264389155 podStartE2EDuration="1.264389155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:13:43.262857908 +0000 UTC m=+2425.637181734" watchObservedRunningTime="2026-09-09 22:13:43.264389155 +0000 UTC m=+2425.638712979"
Sep 09 22:13:52.801489 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:52.801442    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:52.801489 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:52.801503    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:52.814709 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:52.814680    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:13:53.305904 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:13:53.305877    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:14:18.079778 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:18.079742    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:14:19.248161 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:19.248123    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:14:31.233786 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:31.233745    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:14:32.084666 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:32.084629    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"]
Sep 09 22:14:32.107950 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:32.107917    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:14:38.325180 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:38.325137    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:14:46.202115 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.202070    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"]
Sep 09 22:14:46.205434 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.205404    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.217631 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.217603    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"]
Sep 09 22:14:46.340950 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.340910    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f518d4f-a0d4-46c4-a8da-4903132904a2-metrics-certs\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.341173 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.340995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6kb\" (UniqueName: \"kubernetes.io/projected/7f518d4f-a0d4-46c4-a8da-4903132904a2-kube-api-access-ss6kb\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.341173 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.341043    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f518d4f-a0d4-46c4-a8da-4903132904a2-openshift-service-ca\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.341173 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.341086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f518d4f-a0d4-46c4-a8da-4903132904a2-maas-api-tls\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.442125 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.442094    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f518d4f-a0d4-46c4-a8da-4903132904a2-metrics-certs\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.442125 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.442131    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6kb\" (UniqueName: \"kubernetes.io/projected/7f518d4f-a0d4-46c4-a8da-4903132904a2-kube-api-access-ss6kb\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.442404 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.442163    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f518d4f-a0d4-46c4-a8da-4903132904a2-openshift-service-ca\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.442404 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.442191    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f518d4f-a0d4-46c4-a8da-4903132904a2-maas-api-tls\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.442781 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.442761    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f518d4f-a0d4-46c4-a8da-4903132904a2-openshift-service-ca\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.444870 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.444818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f518d4f-a0d4-46c4-a8da-4903132904a2-metrics-certs\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.445157 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.445130    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f518d4f-a0d4-46c4-a8da-4903132904a2-maas-api-tls\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.451467 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.451440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6kb\" (UniqueName: \"kubernetes.io/projected/7f518d4f-a0d4-46c4-a8da-4903132904a2-kube-api-access-ss6kb\") pod \"maas-api-6c5bd889d8-bq6kf\" (UID: \"7f518d4f-a0d4-46c4-a8da-4903132904a2\") " pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.517473 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.517364    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:46.791875 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:46.791753    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"]
Sep 09 22:14:46.794004 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:14:46.793979    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f518d4f_a0d4_46c4_a8da_4903132904a2.slice/crio-2f7f5844da5dbe7630f892434271a2cb6bc78c3e9fea4279856a6ef97432603b WatchSource:0}: Error finding container 2f7f5844da5dbe7630f892434271a2cb6bc78c3e9fea4279856a6ef97432603b: Status 404 returned error can't find the container with id 2f7f5844da5dbe7630f892434271a2cb6bc78c3e9fea4279856a6ef97432603b
Sep 09 22:14:47.105380 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.105266    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml" podUID="e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" containerName="main" containerID="cri-o://db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e" gracePeriod=45
Sep 09 22:14:47.344992 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.344963    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:14:47.351291 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351267    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tls-certs\") pod \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") "
Sep 09 22:14:47.351418 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351311    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-model-cache\") pod \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") "
Sep 09 22:14:47.351418 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351385    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-dshm\") pod \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") "
Sep 09 22:14:47.351418 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351414    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tmp-dir\") pod \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") "
Sep 09 22:14:47.351579 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351436    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hctv\" (UniqueName: \"kubernetes.io/projected/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-kube-api-access-2hctv\") pod \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") "
Sep 09 22:14:47.351579 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351469    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-home\") pod \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\" (UID: \"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0\") "
Sep 09 22:14:47.351695 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351620    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" (UID: "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:14:47.351750 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351727    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" (UID: "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:14:47.351799 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351750    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:47.351882 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.351855    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-home" (OuterVolumeSpecName: "home") pod "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" (UID: "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:14:47.353964 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.353940    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" (UID: "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:14:47.353964 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.353955    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-dshm" (OuterVolumeSpecName: "dshm") pod "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" (UID: "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:14:47.354101 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.353965    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-kube-api-access-2hctv" (OuterVolumeSpecName: "kube-api-access-2hctv") pod "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" (UID: "e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0"). InnerVolumeSpecName "kube-api-access-2hctv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:14:47.452347 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.452313    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:47.452477 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.452351    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:47.452477 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.452366    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hctv\" (UniqueName: \"kubernetes.io/projected/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-kube-api-access-2hctv\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:47.452477 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.452379    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:47.452477 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.452391    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:47.549248 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.549188    2585 generic.go:356] "Generic (PLEG): container finished" podID="e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" containerID="db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e" exitCode=0
Sep 09 22:14:47.549463 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.549257    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"
Sep 09 22:14:47.549463 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.549282    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml" event={"ID":"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0","Type":"ContainerDied","Data":"db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e"}
Sep 09 22:14:47.549463 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.549332    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml" event={"ID":"e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0","Type":"ContainerDied","Data":"1f5636d66c8a25a5fbec0236e6d0d1842ce8b4c6b0fc227073025a4c9c629e65"}
Sep 09 22:14:47.549463 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.549351    2585 scope.go:117] "RemoveContainer" containerID="db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e"
Sep 09 22:14:47.551169 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.551144    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf" event={"ID":"7f518d4f-a0d4-46c4-a8da-4903132904a2","Type":"ContainerStarted","Data":"9062b407d6f1eeb899c4924a04ac00a5ae3233017c4c81ed24285f327f9d6936"}
Sep 09 22:14:47.551310 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.551194    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf" event={"ID":"7f518d4f-a0d4-46c4-a8da-4903132904a2","Type":"ContainerStarted","Data":"2f7f5844da5dbe7630f892434271a2cb6bc78c3e9fea4279856a6ef97432603b"}
Sep 09 22:14:47.558972 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.558955    2585 scope.go:117] "RemoveContainer" containerID="db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e"
Sep 09 22:14:47.559303 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:14:47.559283    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e\": container with ID starting with db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e not found: ID does not exist" containerID="db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e"
Sep 09 22:14:47.559370 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.559313    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e"} err="failed to get container status \"db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e\": rpc error: code = NotFound desc = could not find container \"db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e\": container with ID starting with db5447add19497c0449e9d715f97c8c78a1a7502c3058fd9798a8bdb9956a34e not found: ID does not exist"
Sep 09 22:14:47.570255 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.570195    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf" podStartSLOduration=1.150243739 podStartE2EDuration="1.570181779s" totalImagesPullingTime="419.93804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:14:47.569161837 +0000 UTC m=+2489.943485652" watchObservedRunningTime="2026-09-09 22:14:47.570181779 +0000 UTC m=+2489.944505603"
Sep 09 22:14:47.585877 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.585852    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"]
Sep 09 22:14:47.590554 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:47.590528    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fcc30655-kserve-cb4479b9f-76qml"]
Sep 09 22:14:48.130254 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:48.130194    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" path="/var/lib/kubelet/pods/e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0/volumes"
Sep 09 22:14:48.557144 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:48.557113    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:54.567693 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:54.567659    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c5bd889d8-bq6kf"
Sep 09 22:14:54.617939 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:54.617905    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"]
Sep 09 22:14:54.618200 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:54.618170    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk" podUID="b7356283-c0da-491b-879f-4db48a985a01" containerName="maas-api" containerID="cri-o://5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1" gracePeriod=30
Sep 09 22:14:55.864070 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.864045    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 22:14:55.928654 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.928618    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7356283-c0da-491b-879f-4db48a985a01-openshift-service-ca\") pod \"b7356283-c0da-491b-879f-4db48a985a01\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") "
Sep 09 22:14:55.928859 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.928692    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj84m\" (UniqueName: \"kubernetes.io/projected/b7356283-c0da-491b-879f-4db48a985a01-kube-api-access-jj84m\") pod \"b7356283-c0da-491b-879f-4db48a985a01\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") "
Sep 09 22:14:55.928859 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.928713    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-maas-api-tls\") pod \"b7356283-c0da-491b-879f-4db48a985a01\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") "
Sep 09 22:14:55.928859 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.928747    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-metrics-certs\") pod \"b7356283-c0da-491b-879f-4db48a985a01\" (UID: \"b7356283-c0da-491b-879f-4db48a985a01\") "
Sep 09 22:14:55.929080 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.929051    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7356283-c0da-491b-879f-4db48a985a01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7356283-c0da-491b-879f-4db48a985a01" (UID: "b7356283-c0da-491b-879f-4db48a985a01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:14:55.930974 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.930946    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7356283-c0da-491b-879f-4db48a985a01" (UID: "b7356283-c0da-491b-879f-4db48a985a01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:14:55.931171 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.930999    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b7356283-c0da-491b-879f-4db48a985a01" (UID: "b7356283-c0da-491b-879f-4db48a985a01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:14:55.931171 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:55.931097    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7356283-c0da-491b-879f-4db48a985a01-kube-api-access-jj84m" (OuterVolumeSpecName: "kube-api-access-jj84m") pod "b7356283-c0da-491b-879f-4db48a985a01" (UID: "b7356283-c0da-491b-879f-4db48a985a01"). InnerVolumeSpecName "kube-api-access-jj84m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:14:56.030146 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.030052    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7356283-c0da-491b-879f-4db48a985a01-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:56.030146 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.030083    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj84m\" (UniqueName: \"kubernetes.io/projected/b7356283-c0da-491b-879f-4db48a985a01-kube-api-access-jj84m\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:56.030146 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.030093    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:56.030146 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.030103    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7356283-c0da-491b-879f-4db48a985a01-metrics-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Sep 09 22:14:56.597982 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.597940    2585 generic.go:356] "Generic (PLEG): container finished" podID="b7356283-c0da-491b-879f-4db48a985a01" containerID="5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1" exitCode=0
Sep 09 22:14:56.598190 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.597991    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"
Sep 09 22:14:56.598190 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.598030    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk" event={"ID":"b7356283-c0da-491b-879f-4db48a985a01","Type":"ContainerDied","Data":"5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1"}
Sep 09 22:14:56.598190 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.598086    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk" event={"ID":"b7356283-c0da-491b-879f-4db48a985a01","Type":"ContainerDied","Data":"c2779d0c7aec89f1c82b3b83a33564533944ef58af996d4b8ff9a375a363d100"}
Sep 09 22:14:56.598190 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.598110    2585 scope.go:117] "RemoveContainer" containerID="5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1"
Sep 09 22:14:56.606933 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.606913    2585 scope.go:117] "RemoveContainer" containerID="5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1"
Sep 09 22:14:56.607187 ip-10-0-128-94 kubenswrapper[2585]: E0909 22:14:56.607167    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1\": container with ID starting with 5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1 not found: ID does not exist" containerID="5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1"
Sep 09 22:14:56.607268 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.607197    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1"} err="failed to get container status \"5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1\": rpc error: code = NotFound desc = could not find container \"5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1\": container with ID starting with 5b7490296f6a14949e405fb602506aebd4fb8d182b43b3e53d226f04197313b1 not found: ID does not exist"
Sep 09 22:14:56.616017 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.615989    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"]
Sep 09 22:14:56.619864 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:56.619843    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c59f7b5b-88sjk"]
Sep 09 22:14:58.127628 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:14:58.127590    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7356283-c0da-491b-879f-4db48a985a01" path="/var/lib/kubelet/pods/b7356283-c0da-491b-879f-4db48a985a01/volumes"
Sep 09 22:15:22.309990 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:22.309949    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4f964485-z6vhn_fe266c2f-60ed-4cfb-86a4-d9b202c0232d/manager/0.log"
Sep 09 22:15:23.203606 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:23.203576    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fbbd4d59-xwsjp_844f6c61-76af-4b00-a030-49d052aa4cd8/copy-manifests/0.log"
Sep 09 22:15:23.211411 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:23.211381    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fbbd4d59-xwsjp_844f6c61-76af-4b00-a030-49d052aa4cd8/manager/0.log"
Sep 09 22:15:23.581810 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:23.581719    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c5bd889d8-bq6kf_7f518d4f-a0d4-46c4-a8da-4903132904a2/maas-api/0.log"
Sep 09 22:15:25.185063 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:25.185027    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-42b8p_6f6ba88e-9177-4b70-b6c5-f8039ed66887/manager/0.log"
Sep 09 22:15:25.402255 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:25.402216    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:15:25.558383 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:25.558299    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hmktt_19341b3e-e27a-432f-8d22-22b4c344ff9c/manager/0.log"
Sep 09 22:15:25.840162 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:25.840078    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-mjskk_a4ebf895-8937-4ca0-9144-66c3303a7e39/manager/0.log"
Sep 09 22:15:26.411913 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:26.411885    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xzdss_dc34e697-7c3d-4685-9636-b6134b5ba4f5/discovery/0.log"
Sep 09 22:15:26.648359 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:26.648323    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5769888876-m84dl_50d49099-98f8-4bfe-bdce-4a90f8abfcb7/kube-auth-proxy/0.log"
Sep 09 22:15:26.938936 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:26.938890    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6wpd7_b2990b2d-f023-4113-a7b2-998167d03da9/payload-pre-processing/0.log"
Sep 09 22:15:27.588354 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:27.588322    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-s2rnh_10fc3502-1265-4646-be1a-69a52282ab45/main/0.log"
Sep 09 22:15:28.230495 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:28.230460    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lklhb_528eab46-49f0-42cf-b7b7-cd837198e7d5/main/0.log"
Sep 09 22:15:34.783804 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:34.783767    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:15:34.855894 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:34.855858    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:15:38.031748 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:38.031716    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mgvxv_e2017749-1a67-4077-8ae7-126c4fa84cf2/global-pull-secret-syncer/0.log"
Sep 09 22:15:38.088964 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:38.088930    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log"
Sep 09 22:15:38.178008 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:38.177976    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-94.ec2.internal_fcde823c27ff5664b9e4fd7c2bf072bf/haproxy/0.log"
Sep 09 22:15:42.539274 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:42.539237    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-42b8p_6f6ba88e-9177-4b70-b6c5-f8039ed66887/manager/0.log"
Sep 09 22:15:42.662225 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:42.662158    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hmktt_19341b3e-e27a-432f-8d22-22b4c344ff9c/manager/0.log"
Sep 09 22:15:42.773488 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:42.773453    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-mjskk_a4ebf895-8937-4ca0-9144-66c3303a7e39/manager/0.log"
Sep 09 22:15:44.473395 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:44.473357    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-ghdpm_5b6bbb5c-ca8d-41f8-8d24-944c0848b0ac/cluster-monitoring-operator/0.log"
Sep 09 22:15:44.790410 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:44.790382    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxs2p_eacb2e6e-8c75-4f97-a7eb-297d04c695ac/node-exporter/0.log"
Sep 09 22:15:44.813987 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:44.813959    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxs2p_eacb2e6e-8c75-4f97-a7eb-297d04c695ac/kube-rbac-proxy/0.log"
Sep 09 22:15:44.830745 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:44.830722    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxs2p_eacb2e6e-8c75-4f97-a7eb-297d04c695ac/init-textfile/0.log"
Sep 09 22:15:44.964982 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:44.964952    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0f5225f-c0ed-444c-8c09-007a3c47448a/prometheus/0.log"
Sep 09 22:15:44.987918 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:44.987887    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0f5225f-c0ed-444c-8c09-007a3c47448a/config-reloader/0.log"
Sep 09 22:15:45.010879 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:45.010851    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0f5225f-c0ed-444c-8c09-007a3c47448a/thanos-sidecar/0.log"
Sep 09 22:15:45.029687 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:45.029613    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0f5225f-c0ed-444c-8c09-007a3c47448a/kube-rbac-proxy-web/0.log"
Sep 09 22:15:45.051834 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:45.051804    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0f5225f-c0ed-444c-8c09-007a3c47448a/kube-rbac-proxy/0.log"
Sep 09 22:15:45.074379 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:45.074349    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0f5225f-c0ed-444c-8c09-007a3c47448a/kube-rbac-proxy-thanos/0.log"
Sep 09 22:15:45.102194 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:45.102169    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0f5225f-c0ed-444c-8c09-007a3c47448a/init-config-reloader/0.log"
Sep 09 22:15:48.322118 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.322078    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"]
Sep 09 22:15:48.322520 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.322439    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" containerName="main"
Sep 09 22:15:48.322520 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.322450    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" containerName="main"
Sep 09 22:15:48.322520 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.322467    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7356283-c0da-491b-879f-4db48a985a01" containerName="maas-api"
Sep 09 22:15:48.322520 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.322472    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7356283-c0da-491b-879f-4db48a985a01" containerName="maas-api"
Sep 09 22:15:48.322651 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.322524    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9c5cd64-e565-47a9-b55f-5f6a9c2f8bb0" containerName="main"
Sep 09 22:15:48.322651 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.322534    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7356283-c0da-491b-879f-4db48a985a01" containerName="maas-api"
Sep 09 22:15:48.325825 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.325805    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.328460 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.328438    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j2gqd\"/\"openshift-service-ca.crt\""
Sep 09 22:15:48.328592 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.328439    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j2gqd\"/\"default-dockercfg-7b6pt\""
Sep 09 22:15:48.329102 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.329088    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j2gqd\"/\"kube-root-ca.crt\""
Sep 09 22:15:48.335230 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.335194    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"]
Sep 09 22:15:48.408934 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.408900    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhnb5\" (UniqueName: \"kubernetes.io/projected/9a801c1c-29de-40d7-9263-edfae2cbc10a-kube-api-access-rhnb5\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.408934 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.408932    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-sys\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.409193 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.408980    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-proc\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.409193 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.409034    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-lib-modules\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.409193 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.409068    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-podres\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510041 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510002    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-proc\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510302 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510067    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-lib-modules\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510302 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510099    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-podres\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510302 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510129    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-proc\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510302 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhnb5\" (UniqueName: \"kubernetes.io/projected/9a801c1c-29de-40d7-9263-edfae2cbc10a-kube-api-access-rhnb5\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510302 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510180    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-sys\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510302 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510200    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-lib-modules\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510302 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510247    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-podres\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.510302 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.510286    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a801c1c-29de-40d7-9263-edfae2cbc10a-sys\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.519365 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.519339    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhnb5\" (UniqueName: \"kubernetes.io/projected/9a801c1c-29de-40d7-9263-edfae2cbc10a-kube-api-access-rhnb5\") pod \"perf-node-gather-daemonset-29wr9\" (UID: \"9a801c1c-29de-40d7-9263-edfae2cbc10a\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.636400 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.636301    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:48.908012 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:48.907975    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"]
Sep 09 22:15:48.911299 ip-10-0-128-94 kubenswrapper[2585]: W0909 22:15:48.911269    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9a801c1c_29de_40d7_9263_edfae2cbc10a.slice/crio-a15b585cb1dd06027795e4a35f5c208d538d008755f037ba304e912bf1e1e729 WatchSource:0}: Error finding container a15b585cb1dd06027795e4a35f5c208d538d008755f037ba304e912bf1e1e729: Status 404 returned error can't find the container with id a15b585cb1dd06027795e4a35f5c208d538d008755f037ba304e912bf1e1e729
Sep 09 22:15:49.854543 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:49.854506    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9" event={"ID":"9a801c1c-29de-40d7-9263-edfae2cbc10a","Type":"ContainerStarted","Data":"db58df5455c750123066c3f6831628149b9508591af104131fbe427b3a7e5390"}
Sep 09 22:15:49.855022 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:49.854563    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9" event={"ID":"9a801c1c-29de-40d7-9263-edfae2cbc10a","Type":"ContainerStarted","Data":"a15b585cb1dd06027795e4a35f5c208d538d008755f037ba304e912bf1e1e729"}
Sep 09 22:15:49.855022 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:49.854617    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:49.870145 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:49.870097    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9" podStartSLOduration=1.870082048 podStartE2EDuration="1.870082048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:15:49.868759352 +0000 UTC m=+2552.243083185" watchObservedRunningTime="2026-09-09 22:15:49.870082048 +0000 UTC m=+2552.244405871"
Sep 09 22:15:50.197749 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:50.197720    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sdxrh_b58acc59-65cc-4030-af91-741e1938dc2d/dns/0.log"
Sep 09 22:15:50.216440 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:50.216380    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sdxrh_b58acc59-65cc-4030-af91-741e1938dc2d/kube-rbac-proxy/0.log"
Sep 09 22:15:50.280997 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:50.280966    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rnh76_b875a80b-0dd4-4af6-8222-2f86c18b50a8/dns-node-resolver/0.log"
Sep 09 22:15:50.819069 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:50.819043    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k5kh5_8651d356-a086-46cb-9f76-940a0cee764a/node-ca/0.log"
Sep 09 22:15:52.345648 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:52.345610    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xzdss_dc34e697-7c3d-4685-9636-b6134b5ba4f5/discovery/0.log"
Sep 09 22:15:52.397962 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:52.397928    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5769888876-m84dl_50d49099-98f8-4bfe-bdce-4a90f8abfcb7/kube-auth-proxy/0.log"
Sep 09 22:15:52.469036 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:52.469000    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6wpd7_b2990b2d-f023-4113-a7b2-998167d03da9/payload-pre-processing/0.log"
Sep 09 22:15:53.440600 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:53.440573    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l9tcv_2aee9510-534e-46a2-af83-db4a01b114aa/serve-healthcheck-canary/0.log"
Sep 09 22:15:53.891299 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:53.891219    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-ntfn6_5f17101f-b5eb-4af2-bc07-a05e07df482e/insights-operator/0.log"
Sep 09 22:15:53.891299 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:53.891277    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-ntfn6_5f17101f-b5eb-4af2-bc07-a05e07df482e/insights-operator/1.log"
Sep 09 22:15:53.976602 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:53.976575    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9f6r8_1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866/kube-rbac-proxy/0.log"
Sep 09 22:15:53.995407 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:53.995383    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9f6r8_1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866/exporter/0.log"
Sep 09 22:15:54.015503 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:54.015476    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9f6r8_1ec642b2-ea6f-44ba-8bf1-0d8c0e9c5866/extractor/0.log"
Sep 09 22:15:55.868782 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:55.868755    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-29wr9"
Sep 09 22:15:56.671374 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:56.671336    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4f964485-z6vhn_fe266c2f-60ed-4cfb-86a4-d9b202c0232d/manager/0.log"
Sep 09 22:15:56.974339 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:56.974248    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fbbd4d59-xwsjp_844f6c61-76af-4b00-a030-49d052aa4cd8/manager/0.log"
Sep 09 22:15:56.997843 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:56.997820    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fbbd4d59-xwsjp_844f6c61-76af-4b00-a030-49d052aa4cd8/copy-manifests/0.log"
Sep 09 22:15:58.894022 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:58.893997    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xzdss_dc34e697-7c3d-4685-9636-b6134b5ba4f5/discovery/0.log"
Sep 09 22:15:58.923905 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:58.923873    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5769888876-m84dl_50d49099-98f8-4bfe-bdce-4a90f8abfcb7/kube-auth-proxy/0.log"
Sep 09 22:15:58.989703 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:15:58.989667    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6wpd7_b2990b2d-f023-4113-a7b2-998167d03da9/payload-pre-processing/0.log"
Sep 09 22:16:01.158775 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:01.158740    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xzdss_dc34e697-7c3d-4685-9636-b6134b5ba4f5/discovery/0.log"
Sep 09 22:16:01.196669 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:01.196639    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5769888876-m84dl_50d49099-98f8-4bfe-bdce-4a90f8abfcb7/kube-auth-proxy/0.log"
Sep 09 22:16:01.254928 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:01.254828    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6wpd7_b2990b2d-f023-4113-a7b2-998167d03da9/payload-pre-processing/0.log"
Sep 09 22:16:03.240495 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:03.240445    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:16:03.514165 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:03.514079    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kjxps_905fe570-9725-4238-a36e-9991b7c5ee97/migrator/0.log"
Sep 09 22:16:03.533413 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:03.533387    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kjxps_905fe570-9725-4238-a36e-9991b7c5ee97/graceful-termination/0.log"
Sep 09 22:16:04.397306 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:04.397271    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsd9r_02af621e-6be7-4290-982f-8b6ee14bfd8c/konnectivity-agent/0.log" limit=1024
Sep 09 22:16:05.168737 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.168707    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4dct_03550965-bdc0-4bfe-8915-b27d8e65b0d8/kube-multus-additional-cni-plugins/0.log"
Sep 09 22:16:05.187851 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.187824    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4dct_03550965-bdc0-4bfe-8915-b27d8e65b0d8/egress-router-binary-copy/0.log"
Sep 09 22:16:05.207354 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.207326    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4dct_03550965-bdc0-4bfe-8915-b27d8e65b0d8/cni-plugins/0.log"
Sep 09 22:16:05.231316 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.231294    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4dct_03550965-bdc0-4bfe-8915-b27d8e65b0d8/bond-cni-plugin/0.log"
Sep 09 22:16:05.253262 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.253239    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4dct_03550965-bdc0-4bfe-8915-b27d8e65b0d8/routeoverride-cni/0.log"
Sep 09 22:16:05.274979 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.274955    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4dct_03550965-bdc0-4bfe-8915-b27d8e65b0d8/whereabouts-cni-bincopy/0.log"
Sep 09 22:16:05.297428 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.297392    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4dct_03550965-bdc0-4bfe-8915-b27d8e65b0d8/whereabouts-cni/0.log"
Sep 09 22:16:05.355719 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.355682    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b9qzc_1395e458-dd6f-4c0c-9fdd-bd31989597de/kube-multus/0.log"
Sep 09 22:16:05.476366 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.476283    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qvhdr_b4059f83-eb9e-4c96-b934-0fb98b2716da/network-metrics-daemon/0.log"
Sep 09 22:16:05.492554 ip-10-0-128-94 kubenswrapper[2585]: I0909 22:16:05.492528    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qvhdr_b4059f83-eb9e-4c96-b934-0fb98b2716da/kube-rbac-proxy/0.log"