Sep 08 18:33:59.529703 ip-10-0-140-26 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 18:33:59.939350 ip-10-0-140-26 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:33:59.939350 ip-10-0-140-26 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 18:33:59.939350 ip-10-0-140-26 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:33:59.939350 ip-10-0-140-26 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 18:33:59.939350 ip-10-0-140-26 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:33:59.940363 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.940260    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 18:33:59.946604 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946576    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:33:59.946604 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946596    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:33:59.946604 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946599    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:33:59.946604 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946602    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:33:59.946604 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946606    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:33:59.946604 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946609    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:33:59.946604 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946612    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:33:59.946604 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946615    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946618    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946621    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946624    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946626    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946629    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946632    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946634    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946639    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946641    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946644    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946647    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946649    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946652    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946655    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946657    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946660    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946663    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946667    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:33:59.946897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946671    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946674    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946677    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946680    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946683    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946687    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946689    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946692    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946695    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946697    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946700    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946702    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946705    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946708    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946710    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946713    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946716    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946719    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946722    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946724    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:33:59.947379 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946727    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946730    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946732    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946735    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946737    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946740    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946743    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946745    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946748    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946750    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946753    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946758    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946761    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946763    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946766    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946769    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946771    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946774    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946776    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946779    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:33:59.947871 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946782    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946784    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946787    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946791    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946795    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946798    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946801    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946804    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946807    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946812    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946816    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946819    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946822    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946825    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946828    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946831    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946834    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946837    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946840    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:33:59.948373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946843    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946845    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946848    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946851    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946854    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946857    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946860    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946863    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946866    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946869    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946871    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946874    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946877    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.946880    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947626    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947634    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947637    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947640    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947644    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947647    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:33:59.948832 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947650    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947652    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947655    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947658    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947660    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947663    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947666    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947669    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947673    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947676    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947678    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947682    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947684    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947687    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947689    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947692    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947696    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947700    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947703    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:33:59.949313 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947705    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947708    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947711    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947714    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947717    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947720    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947722    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947725    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947727    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947730    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947732    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947736    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947740    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947742    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947745    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947748    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947750    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947753    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947755    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947758    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:33:59.949829 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947760    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947763    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947766    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947770    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947774    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947777    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947780    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947783    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947785    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947788    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947790    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947793    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947796    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947799    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947802    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947804    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947808    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947811    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947814    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:33:59.950329 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947816    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947819    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947822    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947824    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947828    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947831    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947833    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947836    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947839    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947841    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947844    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947846    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947849    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947852    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947854    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947857    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947859    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947862    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947864    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947867    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:33:59.950802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947870    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947874    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947876    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947879    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947882    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947884    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947887    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947889    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947892    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947894    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947897    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947899    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947902    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947904    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.947907    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948331    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948352    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948357    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948362    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948366    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:33:59.951289 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948370    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948372    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948375    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948380    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948383    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948386    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948390    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948394    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948397    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948399    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948402    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948406    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948409    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948411    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948414    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948416    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948419    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948422    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948424    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:33:59.951798 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948427    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948429    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948432    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948434    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948437    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948439    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948442    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948444    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948447    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948450    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948452    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948455    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948458    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948461    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948464    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948466    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948469    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948471    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948474    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948476    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:33:59.952285 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948479    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948481    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948484    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948486    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948489    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948492    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948495    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948497    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948501    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948503    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948506    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948508    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948511    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948514    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948516    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948519    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948521    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948524    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948526    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948528    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:33:59.952792 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948533    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948537    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948541    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948546    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948548    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948551    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948554    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948556    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948559    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948561    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948564    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948567    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948570    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948572    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948574    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948577    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948579    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948582    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948585    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:33:59.953281 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948588    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948590    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948593    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948595    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948597    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948600    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948602    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948605    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948607    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948610    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948612    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948615    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948617    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948620    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948623    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.948626    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948700    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948738    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948745    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948749    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948754    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 18:33:59.953763 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948757    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948762    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948766    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948769    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948772    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948775    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948779    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948782    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948784    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948787    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948790    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948794    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948797    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948801    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948804    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948807    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948810    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948813    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948817    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948820    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948823    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948826    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948829    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948832    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 18:33:59.954282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948835    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948841    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948844    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948849    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948852    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948855    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948858    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948861    2579 flags.go:64] FLAG: --enable-server="true"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948864    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948869    2579 flags.go:64] FLAG: --event-burst="100"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948872    2579 flags.go:64] FLAG: --event-qps="50"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948875    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948878    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948881    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948884    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948887    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948890    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948893    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948896    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948899    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948902    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948905    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948908    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948910    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948913    2579 flags.go:64] FLAG: --feature-gates=""
Sep 08 18:33:59.954905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948917    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948920    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948923    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948926    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948929    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948932    2579 flags.go:64] FLAG: --help="false"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948935    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-26.ec2.internal"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948938    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948942    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948945    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948950    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948953    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948956    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948959    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948962    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948965    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948967    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948970    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948973    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948976    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948983    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948986    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948989    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948992    2579 flags.go:64] FLAG: --lock-file=""
Sep 08 18:33:59.955688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948994    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.948997    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949001    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949006    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949010    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949013    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949016    2579 flags.go:64] FLAG: --logging-format="text"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949019    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949022    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949025    2579 flags.go:64] FLAG: --manifest-url=""
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949027    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949032    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949034    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949039    2579 flags.go:64] FLAG: --max-pods="110"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949041    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949044    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949049    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949051    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949056    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949059    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949062    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949070    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949074    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949077    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 18:33:59.956740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949080    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949082    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949088    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949091    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949094    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949096    2579 flags.go:64] FLAG: --port="10250"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949099    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949103    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-060b90773393008a1"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949106    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949109    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949112    2579 flags.go:64] FLAG: --register-node="true"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949115    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949118    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949122    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949124    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949127    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949131    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949134    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949137    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949139    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949142    2579 flags.go:64] FLAG: --runonce="false"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949145    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949148    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949151    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949153    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 18:33:59.957790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949159    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949163    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949166    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949169    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949172    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949175    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949178    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949180    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949183    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949186    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949189    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949195    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949198    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949200    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949205    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949207    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949210    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949213    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949216    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949219    2579 flags.go:64] FLAG: --v="2"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949223    2579 flags.go:64] FLAG: --version="false"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949227    2579 flags.go:64] FLAG: --vmodule=""
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949231    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.949235    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949377    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:33:59.958417 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949382    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949385    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949388    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949391    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949394    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949397    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949400    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949404    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949407    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949411    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949413    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949416    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949419    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949421    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949424    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949427    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949429    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949432    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949434    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:33:59.959063 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949437    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949440    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949442    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949445    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949448    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949450    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949453    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949455    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949458    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949461    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949464    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949467    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949469    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949472    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949475    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949477    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949480    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949482    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949485    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949488    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:33:59.959552 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949492    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949497    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949501    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949504    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949507    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949510    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949512    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949515    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949518    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949521    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949523    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949526    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949528    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949531    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949533    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949536    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949538    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949541    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949543    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:33:59.960041 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949546    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949548    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949551    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949553    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949556    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949560    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949563    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949566    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949569    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949572    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949574    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949577    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949579    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949584    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949588    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949590    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949593    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949596    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949598    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949601    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:33:59.960533 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949603    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949606    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949608    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949612    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949616    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949618    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949621    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949623    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949626    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949629    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949632    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949634    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949636    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949639    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949641    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949644    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949647    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949649    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949652    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:33:59.961265 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.949654    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.950120    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.957676    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.957701    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.960879    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.960902    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.960996    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961007    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961012    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961017    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961021    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961025    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961029    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961032    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961036    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:33:59.962242 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961040    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961044    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961048    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961052    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961057    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961061    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961065    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961069    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961073    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961077    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961083    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961088    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961092    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961096    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961101    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961105    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961110    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961114    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961119    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:33:59.962727 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961125    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961132    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961138    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961143    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961148    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961153    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961157    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961162    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961166    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961170    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961174    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961178    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961182    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961187    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961191    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961195    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961199    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961203    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961208    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961213    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:33:59.963456 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961217    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961221    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961226    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961230    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961234    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961238    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961244    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961248    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961252    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961256    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961260    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961264    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961268    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961272    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961278    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961282    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961287    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961292    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961296    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961300    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:33:59.964377 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961304    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961309    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961313    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961317    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961322    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961327    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961331    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961335    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961355    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961359    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961363    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961368    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961372    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961377    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961380    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961384    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961388    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961392    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961396    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961400    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:33:59.964939 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961403    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961409    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961415    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961419    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961423    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961428    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961432    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961437    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961441    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961445    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961449    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.961457    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961645    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961653    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961658    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:33:59.965593 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961663    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961668    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961673    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961678    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961682    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961686    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961690    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961695    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961699    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961704    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961709    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961713    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961717    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961721    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961726    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961730    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961734    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961738    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961741    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961745    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961750    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:33:59.965978 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961754    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961758    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961762    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961767    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961772    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961776    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961780    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961785    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961789    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961793    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961797    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961802    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961806    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961810    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961814    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961818    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961822    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961826    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961830    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:33:59.966553 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961835    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961839    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961843    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961847    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961851    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961856    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961860    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961866    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961872    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961878    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961882    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961887    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961891    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961895    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961899    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961903    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961907    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961911    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961916    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961921    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:33:59.967079 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961925    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961929    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961933    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961939    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961945    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961950    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961954    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961958    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961962    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961966    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961970    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961975    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961980    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961984    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961989    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961993    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.961997    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962001    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962005    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:33:59.967612 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962009    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962013    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962018    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962022    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962026    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962031    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962035    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962039    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962046    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962051    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962056    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962060    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962065    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962070    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962074    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962078    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:33:59.968073 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:33:59.962082    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:33:59.968761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.962089    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:33:59.968761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.962290    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 18:33:59.968761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.964954    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 18:33:59.968761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.966044    2579 server.go:1019] "Starting client certificate rotation"
Sep 08 18:33:59.968761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.966151    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 18:33:59.968761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.966188    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 18:33:59.969009 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.968788    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 18:33:59.970096 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.970060    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 18:33:59.981915 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.981896    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 08 18:33:59.984764 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.984744    2579 log.go:25] "Validated CRI v1 image API"
Sep 08 18:33:59.985699 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.985683    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 18:33:59.988845 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.988824    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c670410c-0593-4342-9e41-ef49a58e6bce:/dev/nvme0n1p4 f0d6a09a-a619-49ff-8dda-886cddfa93ca:/dev/nvme0n1p3]
Sep 08 18:33:59.988921 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.988844    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 18:33:59.994025 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.993906    2579 manager.go:216] Machine: {Timestamp:2026-09-08 18:33:59.992510398 +0000 UTC m=+0.346603722 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105079 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eb66155f06ca4f16577d0e93a0dfc SystemUUID:ec2eb661-55f0-6ca4-f165-77d0e93a0dfc BootID:941034df-a316-4f53-a93c-23c48a936f7b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f2:7a:32:9a:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f2:7a:32:9a:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:5f:9a:c0:2b:3f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 18:33:59.994025 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.994013    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 18:33:59.994191 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.994121    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 18:33:59.995430 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.995414    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 18:33:59.997204 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.997180    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 18:33:59.997355 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.997208    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 18:33:59.997936 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.997926    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 18:33:59.997967 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.997938    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 18:33:59.997967 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.997945    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 18:33:59.998638 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.998627    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 18:33:59.999397 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:33:59.999387    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 18:34:00.002057 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002046    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 18:34:00.002187 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002178    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 18:34:00.002224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002208    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 18:34:00.002224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002218    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 18:34:00.002290 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002233    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 18:34:00.002290 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002242    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 08 18:34:00.002290 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002252    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 18:34:00.002620 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002608    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 18:34:00.002658 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.002629    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 18:34:00.005196 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.005173    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 18:34:00.007139 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.007126    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 18:34:00.007198 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.007156    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 18:34:00.008253 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008240    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 18:34:00.008301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008257    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 18:34:00.008301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008263    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 18:34:00.008301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008269    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 18:34:00.008301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008278    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 18:34:00.008301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008283    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 18:34:00.008301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008296    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 18:34:00.008301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008301    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 18:34:00.008520 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008308    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 18:34:00.008520 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008314    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 18:34:00.008520 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008331    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 18:34:00.008520 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.008336    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 18:34:00.009817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.009799    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 18:34:00.009883 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.009825    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 18:34:00.011931 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.011904    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 18:34:00.011931 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.011917    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 18:34:00.012034 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.012021    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-26.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 18:34:00.013704 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.013683    2579 server.go:1268] "Started kubelet"
Sep 08 18:34:00.013809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.013760    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 18:34:00.013909 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.013761    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 18:34:00.013966 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.013935    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 18:34:00.014722 ip-10-0-140-26 systemd[1]: Started Kubernetes Kubelet.
Sep 08 18:34:00.015810 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.015794    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 18:34:00.018279 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.018266    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 18:34:00.021544 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.021516    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 18:34:00.022974 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.018495    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-26.ec2.internal.18d36c323c653d15  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-26.ec2.internal,UID:ip-10-0-140-26.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-26.ec2.internal,},FirstTimestamp:2026-09-08 18:34:00.013651221 +0000 UTC m=+0.367744545,LastTimestamp:2026-09-08 18:34:00.013651221 +0000 UTC m=+0.367744545,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-26.ec2.internal,}"
Sep 08 18:34:00.023073 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.022981    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 18:34:00.023582 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.023563    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 18:34:00.023779 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.023754    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.024080 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.024068    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 18:34:00.024188 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.024086    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 18:34:00.024273 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.024263    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 18:34:00.024539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.024527    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 18:34:00.024639 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.024629    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 18:34:00.025043 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.025023    2579 factory.go:55] Registering systemd factory
Sep 08 18:34:00.025043 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.025042    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 08 18:34:00.027108 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.027093    2579 factory.go:153] Registering CRI-O factory
Sep 08 18:34:00.027108 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.027111    2579 factory.go:223] Registration of the crio container factory successfully
Sep 08 18:34:00.027241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.027159    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 18:34:00.027241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.027177    2579 factory.go:103] Registering Raw factory
Sep 08 18:34:00.027241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.027194    2579 manager.go:1196] Started watching for new ooms in manager
Sep 08 18:34:00.027907 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.027893    2579 manager.go:349] Starting recovery of all containers
Sep 08 18:34:00.035768 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.035741    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 18:34:00.036286 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.036254    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 18:34:00.038493 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.038477    2579 manager.go:354] Recovery completed
Sep 08 18:34:00.043654 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.043640    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:34:00.045667 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.045635    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:34:00.045761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.045682    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:34:00.045761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.045698    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:34:00.046477 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.046458    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:34:00.046553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.046458    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 18:34:00.046553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.046499    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 18:34:00.046553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.046521    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 18:34:00.047268 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.047251    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:34:00.047377 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.047286    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:34:00.047377 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.047301    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:34:00.047962 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.047949    2579 policy_none.go:49] "None policy: Start"
Sep 08 18:34:00.048007 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.047966    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 18:34:00.048007 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.047977    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 18:34:00.049012 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.049001    2579 policy_none.go:47] "Start"
Sep 08 18:34:00.059842 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.059659    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-26.ec2.internal.18d36c323e4db9be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-26.ec2.internal,UID:ip-10-0-140-26.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-26.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-26.ec2.internal,},FirstTimestamp:2026-09-08 18:34:00.045664702 +0000 UTC m=+0.399758026,LastTimestamp:2026-09-08 18:34:00.045664702 +0000 UTC m=+0.399758026,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-26.ec2.internal,}"
Sep 08 18:34:00.059941 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.059919    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-66kcm"
Sep 08 18:34:00.068488 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.068407    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-26.ec2.internal.18d36c323e4e1b60  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-26.ec2.internal,UID:ip-10-0-140-26.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-26.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-26.ec2.internal,},FirstTimestamp:2026-09-08 18:34:00.045689696 +0000 UTC m=+0.399783022,LastTimestamp:2026-09-08 18:34:00.045689696 +0000 UTC m=+0.399783022,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-26.ec2.internal,}"
Sep 08 18:34:00.070054 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.070036    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-66kcm"
Sep 08 18:34:00.073973 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.073959    2579 manager.go:337] "Starting Device Plugin manager"
Sep 08 18:34:00.074049 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.074001    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 18:34:00.074049 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.074013    2579 server.go:87] "Starting device plugin registration server"
Sep 08 18:34:00.074336 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.074323    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 18:34:00.074532 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.074352    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 18:34:00.074803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.074581    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 18:34:00.074803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.074668    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 18:34:00.074803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.074677    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 18:34:00.075975 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.075954    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 18:34:00.076061 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.076010    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.155038 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.154991    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 18:34:00.156249 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.156219    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 18:34:00.156374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.156274    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 18:34:00.156374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.156299    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 18:34:00.156455 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.156389    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 18:34:00.158741 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.158720    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:34:00.174566 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.174536    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:34:00.175580 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.175559    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:34:00.175674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.175595    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:34:00.175674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.175610    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:34:00.175674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.175639    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.184170 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.184148    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.184262 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.184173    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-26.ec2.internal\": node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.205702 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.205627    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.256741 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.256707    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal"]
Sep 08 18:34:00.256885 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.256782    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:34:00.258205 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.258187    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:34:00.258315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.258224    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:34:00.258315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.258240    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:34:00.259452 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.259438    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:34:00.259587 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.259572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.259649 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.259609    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:34:00.260157 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.260140    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:34:00.260241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.260171    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:34:00.260241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.260140    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:34:00.260241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.260180    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:34:00.260241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.260223    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:34:00.260241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.260237    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:34:00.261365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.261353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.261403 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.261383    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:34:00.262048 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.262034    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:34:00.262135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.262058    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:34:00.262135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.262067    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:34:00.289163 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.289137    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-26.ec2.internal\" not found" node="ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.293670 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.293652    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-26.ec2.internal\" not found" node="ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.306719 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.306677    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.326846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.326822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9036dac6183b2c3d77595670f1fe6a01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal\" (UID: \"9036dac6183b2c3d77595670f1fe6a01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.326959 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.326850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9036dac6183b2c3d77595670f1fe6a01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal\" (UID: \"9036dac6183b2c3d77595670f1fe6a01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.326959 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.326867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e01954d7983c7d343489f8e29f686ea0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-26.ec2.internal\" (UID: \"e01954d7983c7d343489f8e29f686ea0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.407039 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.407001    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.427448 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.427418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9036dac6183b2c3d77595670f1fe6a01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal\" (UID: \"9036dac6183b2c3d77595670f1fe6a01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.427539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.427456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9036dac6183b2c3d77595670f1fe6a01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal\" (UID: \"9036dac6183b2c3d77595670f1fe6a01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.427539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.427512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9036dac6183b2c3d77595670f1fe6a01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal\" (UID: \"9036dac6183b2c3d77595670f1fe6a01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.427539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.427511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9036dac6183b2c3d77595670f1fe6a01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal\" (UID: \"9036dac6183b2c3d77595670f1fe6a01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.427643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.427543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e01954d7983c7d343489f8e29f686ea0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-26.ec2.internal\" (UID: \"e01954d7983c7d343489f8e29f686ea0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.427643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.427588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e01954d7983c7d343489f8e29f686ea0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-26.ec2.internal\" (UID: \"e01954d7983c7d343489f8e29f686ea0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.507925 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.507855    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.594919 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.594878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.597132 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.597115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:00.608266 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.608238    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.713140 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.713085    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.819656 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.819577    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.926734 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:00.926687    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:00.965604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.965566    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 18:34:00.966140 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:00.965754    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 18:34:01.026813 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:01.026779    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:01.074163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.074036    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 18:29:00 +0000 UTC" deadline="2028-03-08 08:44:31.523208732 +0000 UTC"
Sep 08 18:34:01.074163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.074099    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13118h10m30.449115234s"
Sep 08 18:34:01.093325 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.093299    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:34:01.127740 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:01.127711    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:01.219673 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.219633    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:34:01.224138 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.224116    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:34:01.224228 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.224154    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:34:01.224228 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.224168    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:34:01.228689 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:01.228671    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:01.329787 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:01.329677    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:01.430199 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:01.430160    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:01.493566 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:01.493523    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01954d7983c7d343489f8e29f686ea0.slice/crio-fd7a6c0c70d8e2b84fde5734b4330d92a0ff94b20ae63a14c8071ab218834804 WatchSource:0}: Error finding container fd7a6c0c70d8e2b84fde5734b4330d92a0ff94b20ae63a14c8071ab218834804: Status 404 returned error can't find the container with id fd7a6c0c70d8e2b84fde5734b4330d92a0ff94b20ae63a14c8071ab218834804
Sep 08 18:34:01.493716 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:01.493704    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9036dac6183b2c3d77595670f1fe6a01.slice/crio-9a498fb97acddb4cff8cf254a0ced4253ad28fc08a8fc591c648814955ea0a9f WatchSource:0}: Error finding container 9a498fb97acddb4cff8cf254a0ced4253ad28fc08a8fc591c648814955ea0a9f: Status 404 returned error can't find the container with id 9a498fb97acddb4cff8cf254a0ced4253ad28fc08a8fc591c648814955ea0a9f
Sep 08 18:34:01.499185 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.499168    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:34:01.530546 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:01.530511    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-26.ec2.internal\" not found"
Sep 08 18:34:01.535673 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.535651    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:34:01.575564 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.575537    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:34:01.626928 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.626858    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:01.647640 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.647608    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 18:34:01.649760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.649742    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal"
Sep 08 18:34:01.661059 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:01.661036    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 18:34:02.005116 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.005024    2579 apiserver.go:52] "Watching apiserver"
Sep 08 18:34:02.010018 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.009993    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 18:34:02.010434 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.010394    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9b4q4","kube-system/global-pull-secret-syncer-t5hjx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r","openshift-cluster-node-tuning-operator/tuned-tl9d2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal","openshift-multus/multus-bw929","openshift-network-operator/iptables-alerter-zn98k","openshift-ovn-kubernetes/ovnkube-node-m6qmp","kube-system/konnectivity-agent-bqqz7","kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal","openshift-dns/node-resolver-rn2xb","openshift-image-registry/node-ca-kslr8","openshift-multus/multus-additional-cni-plugins-86pss","openshift-multus/network-metrics-daemon-8mlg8"]
Sep 08 18:34:02.012892 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.012874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.014203 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.014179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.014789 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.014768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8vp94\""
Sep 08 18:34:02.014919 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.014902    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 18:34:02.015143 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.015121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 18:34:02.015253 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.015121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:34:02.015585 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.015565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:02.015669 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.015637    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:02.016155 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.016141    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 18:34:02.017141 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.016664    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 18:34:02.017141 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.016964    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x59sx\""
Sep 08 18:34:02.020125 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.018885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.021530 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.021045    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:34:02.021530 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.021232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m7wkk\""
Sep 08 18:34:02.021530 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.021425    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 18:34:02.021872 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.021544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.022750 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.022730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.023514 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.023466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 18:34:02.023514 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.023480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5tdkq\""
Sep 08 18:34:02.023674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.023654    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 18:34:02.023674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.023658    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 18:34:02.023773 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.023689    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 18:34:02.024134 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.024117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:02.024550 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.024526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 18:34:02.024665 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.024627    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 18:34:02.024796 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.024779    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 18:34:02.024965 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.024949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4wgz8\""
Sep 08 18:34:02.025054 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.024991    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 18:34:02.025395 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.025375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.025919 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.025902    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2c64l\""
Sep 08 18:34:02.026022 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.025924    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 18:34:02.026104 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.026074    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 18:34:02.026695 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.026680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:02.026838 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.026783    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:02.027019 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.027002    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6tr75\""
Sep 08 18:34:02.027234 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.027219    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 18:34:02.027428 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.027392    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 18:34:02.028161 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.028147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.028278 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.028262    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:02.030521 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.030505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.031980 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.031964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.032485 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.032469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 18:34:02.032485 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.032480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-44njf\""
Sep 08 18:34:02.032822 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.032805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 18:34:02.032975 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.032958    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 18:34:02.033183 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.033164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 18:34:02.033272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.033171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 18:34:02.033314 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.033278    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 18:34:02.034032 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.034012    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 18:34:02.034098 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.034022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-snghn\""
Sep 08 18:34:02.034503 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.034482    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 18:34:02.034503 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.034492    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 18:34:02.038102 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-cni-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038200 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-cni-binary-copy\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038200 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-cni-bin\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038200 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-hostroot\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038371 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-system-cni-dir\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.038371 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg5rv\" (UniqueName: \"kubernetes.io/projected/eb44a0dc-533c-4750-91aa-766b956e2465-kube-api-access-tg5rv\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.038371 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysctl-d\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.038371 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a37d7aff-75ca-45b8-84ce-38c19e79bb69-tmp\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.038573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-socket-dir-parent\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-cni-multus\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-multus-certs\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lztv\" (UniqueName: \"kubernetes.io/projected/c50318ae-5b05-435f-9ba8-d2552408cdca-kube-api-access-2lztv\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.038573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnzm\" (UniqueName: \"kubernetes.io/projected/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-kube-api-access-djnzm\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.038573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb44a0dc-533c-4750-91aa-766b956e2465-tmp-dir\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.038573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysconfig\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-var-lib-kubelet\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-cnibin\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb44a0dc-533c-4750-91aa-766b956e2465-hosts-file\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-run\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-host\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-system-cni-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-kubelet\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-conf-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.038846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-etc-kubernetes\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-host-slash\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-systemd\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-cnibin\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjvf\" (UniqueName: \"kubernetes.io/projected/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-kube-api-access-lgjvf\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-modprobe-d\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.038995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-kubernetes\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysctl-conf\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwth\" (UniqueName: \"kubernetes.io/projected/a37d7aff-75ca-45b8-84ce-38c19e79bb69-kube-api-access-chwth\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24f743c5-c5d6-47ce-8e49-1865602059a9-agent-certs\") pod \"konnectivity-agent-bqqz7\" (UID: \"24f743c5-c5d6-47ce-8e49-1865602059a9\") " pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-k8s-cni-cncf-io\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-netns\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-cni-binary-copy\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwbs\" (UniqueName: \"kubernetes.io/projected/7a40264b-ad31-4698-926a-5d35143a916f-kube-api-access-2mwbs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:02.039324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-sys\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.039887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-lib-modules\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.039887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-tuned\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.039887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-host\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.039887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-serviceca\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.039887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xjx\" (UniqueName: \"kubernetes.io/projected/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-kube-api-access-f7xjx\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.039887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-daemon-config\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.039887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24f743c5-c5d6-47ce-8e49-1865602059a9-konnectivity-ca\") pod \"konnectivity-agent-bqqz7\" (UID: \"24f743c5-c5d6-47ce-8e49-1865602059a9\") " pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:02.039887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.039522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-iptables-alerter-script\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.040242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.040194    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 18:34:02.059004 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.058978    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ngbz4"
Sep 08 18:34:02.067090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.067060    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ngbz4"
Sep 08 18:34:02.125571 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.125539    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 18:34:02.139788 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.139752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.139788 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.139793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-sys-fs\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.139829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24f743c5-c5d6-47ce-8e49-1865602059a9-konnectivity-ca\") pod \"konnectivity-agent-bqqz7\" (UID: \"24f743c5-c5d6-47ce-8e49-1865602059a9\") " pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.139858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-iptables-alerter-script\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.139928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-cni-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.139958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-cni-binary-copy\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.139984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-slash\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysctl-d\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-cni-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.140090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a37d7aff-75ca-45b8-84ce-38c19e79bb69-tmp\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-cni-multus\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lztv\" (UniqueName: \"kubernetes.io/projected/c50318ae-5b05-435f-9ba8-d2552408cdca-kube-api-access-2lztv\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-var-lib-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb44a0dc-533c-4750-91aa-766b956e2465-tmp-dir\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-cni-multus\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-var-lib-kubelet\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-cnibin\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-var-lib-kubelet\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-systemd\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-log-socket\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-socket-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-system-cni-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-kubelet\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-run-netns\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140478    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-host-slash\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.140618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-iptables-alerter-script\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-system-cni-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-host-slash\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24f743c5-c5d6-47ce-8e49-1865602059a9-konnectivity-ca\") pod \"konnectivity-agent-bqqz7\" (UID: \"24f743c5-c5d6-47ce-8e49-1865602059a9\") " pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-kubelet\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-cnibin\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-systemd\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb44a0dc-533c-4750-91aa-766b956e2465-tmp-dir\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-cnibin\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-systemd\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-cnibin\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysctl-d\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-modprobe-d\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-kubernetes\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwth\" (UniqueName: \"kubernetes.io/projected/a37d7aff-75ca-45b8-84ce-38c19e79bb69-kube-api-access-chwth\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-kubernetes\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-modprobe-d\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-cni-binary-copy\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.141457 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.140982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24f743c5-c5d6-47ce-8e49-1865602059a9-agent-certs\") pod \"konnectivity-agent-bqqz7\" (UID: \"24f743c5-c5d6-47ce-8e49-1865602059a9\") " pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-k8s-cni-cncf-io\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-netns\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-netns\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-k8s-cni-cncf-io\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-cni-binary-copy\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e680fcd6-db0f-49ff-88c8-94c448785e49-kubelet-config\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-ovn\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-cni-netd\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-ovnkube-config\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-tuned\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-host\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-serviceca\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e680fcd6-db0f-49ff-88c8-94c448785e49-dbus\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-cni-bin\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.142462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-hostroot\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-system-cni-dir\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-host\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-node-log\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-var-lib-cni-bin\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-hostroot\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-system-cni-dir\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-cni-bin\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-registration-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg5rv\" (UniqueName: \"kubernetes.io/projected/eb44a0dc-533c-4750-91aa-766b956e2465-kube-api-access-tg5rv\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-cni-binary-copy\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-socket-dir-parent\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-socket-dir-parent\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-multus-certs\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-ovnkube-script-lib\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.143456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnzm\" (UniqueName: \"kubernetes.io/projected/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-kube-api-access-djnzm\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-host-run-multus-certs\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysconfig\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-serviceca\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.141995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysconfig\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-env-overrides\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c50318ae-5b05-435f-9ba8-d2552408cdca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb44a0dc-533c-4750-91aa-766b956e2465-hosts-file\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-run\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-host\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.142367    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-conf-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-run\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-etc-kubernetes\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb44a0dc-533c-4750-91aa-766b956e2465-hosts-file\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.144348 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.142460    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:34:02.642426633 +0000 UTC m=+2.996519959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-etc-kubernetes\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-host\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-conf-dir\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp5sv\" (UniqueName: \"kubernetes.io/projected/4396b3d0-3459-453a-80f8-468b02bee7cb-kube-api-access-xp5sv\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjvf\" (UniqueName: \"kubernetes.io/projected/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-kube-api-access-lgjvf\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4396b3d0-3459-453a-80f8-468b02bee7cb-ovn-node-metrics-cert\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysctl-conf\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-systemd-units\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-etc-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-device-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-etc-selinux\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngn47\" (UniqueName: \"kubernetes.io/projected/db9e740a-de51-49cf-86b7-bbbc23263d53-kube-api-access-ngn47\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwbs\" (UniqueName: \"kubernetes.io/projected/7a40264b-ad31-4698-926a-5d35143a916f-kube-api-access-2mwbs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:02.145131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-sys\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-lib-modules\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xjx\" (UniqueName: \"kubernetes.io/projected/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-kube-api-access-f7xjx\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.142983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-daemon-config\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.143007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-kubelet\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.143109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-sys\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.143133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-sysctl-conf\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.143196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a37d7aff-75ca-45b8-84ce-38c19e79bb69-lib-modules\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.143358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c50318ae-5b05-435f-9ba8-d2552408cdca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.143674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-multus-daemon-config\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.144907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a37d7aff-75ca-45b8-84ce-38c19e79bb69-etc-tuned\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.144945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a37d7aff-75ca-45b8-84ce-38c19e79bb69-tmp\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.146075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.145140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24f743c5-c5d6-47ce-8e49-1865602059a9-agent-certs\") pod \"konnectivity-agent-bqqz7\" (UID: \"24f743c5-c5d6-47ce-8e49-1865602059a9\") " pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:02.148327 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.148307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lztv\" (UniqueName: \"kubernetes.io/projected/c50318ae-5b05-435f-9ba8-d2552408cdca-kube-api-access-2lztv\") pod \"multus-additional-cni-plugins-86pss\" (UID: \"c50318ae-5b05-435f-9ba8-d2552408cdca\") " pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.158747 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.158713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwth\" (UniqueName: \"kubernetes.io/projected/a37d7aff-75ca-45b8-84ce-38c19e79bb69-kube-api-access-chwth\") pod \"tuned-tl9d2\" (UID: \"a37d7aff-75ca-45b8-84ce-38c19e79bb69\") " pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.160128 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.159912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg5rv\" (UniqueName: \"kubernetes.io/projected/eb44a0dc-533c-4750-91aa-766b956e2465-kube-api-access-tg5rv\") pod \"node-resolver-rn2xb\" (UID: \"eb44a0dc-533c-4750-91aa-766b956e2465\") " pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.160128 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.159947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwbs\" (UniqueName: \"kubernetes.io/projected/7a40264b-ad31-4698-926a-5d35143a916f-kube-api-access-2mwbs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:02.160128 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.160035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xjx\" (UniqueName: \"kubernetes.io/projected/21b4eada-cc2e-4655-90b8-38e6f9bbcf6a-kube-api-access-f7xjx\") pod \"node-ca-kslr8\" (UID: \"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a\") " pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.160128 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.160096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnzm\" (UniqueName: \"kubernetes.io/projected/b86a0f2d-bf4a-4070-9e14-00e349e64f8f-kube-api-access-djnzm\") pod \"iptables-alerter-zn98k\" (UID: \"b86a0f2d-bf4a-4070-9e14-00e349e64f8f\") " pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.162404 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.162381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjvf\" (UniqueName: \"kubernetes.io/projected/f05e2e6d-161d-4996-a7c6-e060e84fd1d1-kube-api-access-lgjvf\") pod \"multus-bw929\" (UID: \"f05e2e6d-161d-4996-a7c6-e060e84fd1d1\") " pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.163042 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.162992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal" event={"ID":"9036dac6183b2c3d77595670f1fe6a01","Type":"ContainerStarted","Data":"9a498fb97acddb4cff8cf254a0ced4253ad28fc08a8fc591c648814955ea0a9f"}
Sep 08 18:34:02.164838 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.164813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal" event={"ID":"e01954d7983c7d343489f8e29f686ea0","Type":"ContainerStarted","Data":"fd7a6c0c70d8e2b84fde5734b4330d92a0ff94b20ae63a14c8071ab218834804"}
Sep 08 18:34:02.243616 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-sys-fs\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-slash\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-var-lib-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-sys-fs\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-systemd\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-log-socket\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-socket-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-run-netns\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e680fcd6-db0f-49ff-88c8-94c448785e49-kubelet-config\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.243866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-systemd\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-ovn\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-ovn\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-cni-netd\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-var-lib-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-log-socket\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-slash\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-ovnkube-config\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e680fcd6-db0f-49ff-88c8-94c448785e49-dbus\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.243999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-node-log\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e680fcd6-db0f-49ff-88c8-94c448785e49-kubelet-config\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-cni-bin\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-run-netns\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-registration-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-ovnkube-script-lib\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.244467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-env-overrides\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp5sv\" (UniqueName: \"kubernetes.io/projected/4396b3d0-3459-453a-80f8-468b02bee7cb-kube-api-access-xp5sv\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-cni-bin\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e680fcd6-db0f-49ff-88c8-94c448785e49-dbus\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244383    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-registration-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-node-log\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-socket-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4396b3d0-3459-453a-80f8-468b02bee7cb-ovn-node-metrics-cert\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-env-overrides\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-systemd-units\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-etc-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-device-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-etc-selinux\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.244948    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:02.245316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.244959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngn47\" (UniqueName: \"kubernetes.io/projected/db9e740a-de51-49cf-86b7-bbbc23263d53-kube-api-access-ngn47\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.245004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret podName:e680fcd6-db0f-49ff-88c8-94c448785e49 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:02.744985661 +0000 UTC m=+3.099078994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret") pod "global-pull-secret-syncer-t5hjx" (UID: "e680fcd6-db0f-49ff-88c8-94c448785e49") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-systemd-units\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-ovnkube-config\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-etc-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-cni-netd\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-run-openvswitch\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-device-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4396b3d0-3459-453a-80f8-468b02bee7cb-ovnkube-script-lib\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-kubelet\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4396b3d0-3459-453a-80f8-468b02bee7cb-host-kubelet\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.246181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.245283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db9e740a-de51-49cf-86b7-bbbc23263d53-etc-selinux\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.247722 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.247694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4396b3d0-3459-453a-80f8-468b02bee7cb-ovn-node-metrics-cert\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.251453 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.251430    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:34:02.251578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.251456    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:34:02.251578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.251473    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dxttk for pod openshift-network-diagnostics/network-check-target-9b4q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:02.251578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.251548    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk podName:909a461c-5e63-4736-ad00-247aa33384da nodeName:}" failed. No retries permitted until 2026-09-08 18:34:02.751531087 +0000 UTC m=+3.105624415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dxttk" (UniqueName: "kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk") pod "network-check-target-9b4q4" (UID: "909a461c-5e63-4736-ad00-247aa33384da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:02.254131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.254102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngn47\" (UniqueName: \"kubernetes.io/projected/db9e740a-de51-49cf-86b7-bbbc23263d53-kube-api-access-ngn47\") pod \"aws-ebs-csi-driver-node-h865r\" (UID: \"db9e740a-de51-49cf-86b7-bbbc23263d53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.254936 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.254890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp5sv\" (UniqueName: \"kubernetes.io/projected/4396b3d0-3459-453a-80f8-468b02bee7cb-kube-api-access-xp5sv\") pod \"ovnkube-node-m6qmp\" (UID: \"4396b3d0-3459-453a-80f8-468b02bee7cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.325605 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.325523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zn98k"
Sep 08 18:34:02.333477 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.333191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rn2xb"
Sep 08 18:34:02.341050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.340522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tl9d2"
Sep 08 18:34:02.345854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.345617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kslr8"
Sep 08 18:34:02.353183 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:02.351577    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37d7aff_75ca_45b8_84ce_38c19e79bb69.slice/crio-966908ebaf3af7169593a3aaed81cb4588f417a1352821decbe677960c713284 WatchSource:0}: Error finding container 966908ebaf3af7169593a3aaed81cb4588f417a1352821decbe677960c713284: Status 404 returned error can't find the container with id 966908ebaf3af7169593a3aaed81cb4588f417a1352821decbe677960c713284
Sep 08 18:34:02.353183 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.351741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw929"
Sep 08 18:34:02.353572 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:02.353536    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb44a0dc_533c_4750_91aa_766b956e2465.slice/crio-9c1f1751beb9c0efa08b14df2fc9f806803fbad8c47d072f71f5a6944e49a548 WatchSource:0}: Error finding container 9c1f1751beb9c0efa08b14df2fc9f806803fbad8c47d072f71f5a6944e49a548: Status 404 returned error can't find the container with id 9c1f1751beb9c0efa08b14df2fc9f806803fbad8c47d072f71f5a6944e49a548
Sep 08 18:34:02.358435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.358340    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:02.363143 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.363119    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:34:02.366087 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.366067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86pss"
Sep 08 18:34:02.372081 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:02.372052    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05e2e6d_161d_4996_a7c6_e060e84fd1d1.slice/crio-7b908a61ea5006ff2b9e43403cdfe0c767600ecb8de62ebbc954dc3e986ffef6 WatchSource:0}: Error finding container 7b908a61ea5006ff2b9e43403cdfe0c767600ecb8de62ebbc954dc3e986ffef6: Status 404 returned error can't find the container with id 7b908a61ea5006ff2b9e43403cdfe0c767600ecb8de62ebbc954dc3e986ffef6
Sep 08 18:34:02.373998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.373972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:02.377741 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:02.377717    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f743c5_c5d6_47ce_8e49_1865602059a9.slice/crio-8e5480d3041f055f9ca1bf459d71ff647c41f3d0be2430ffb86f46b530a63cad WatchSource:0}: Error finding container 8e5480d3041f055f9ca1bf459d71ff647c41f3d0be2430ffb86f46b530a63cad: Status 404 returned error can't find the container with id 8e5480d3041f055f9ca1bf459d71ff647c41f3d0be2430ffb86f46b530a63cad
Sep 08 18:34:02.379798 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.379778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r"
Sep 08 18:34:02.391194 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:02.391164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50318ae_5b05_435f_9ba8_d2552408cdca.slice/crio-d09e509f00abb7e14360fe755684ccccfb897dab2395141b03f921ebccd5520a WatchSource:0}: Error finding container d09e509f00abb7e14360fe755684ccccfb897dab2395141b03f921ebccd5520a: Status 404 returned error can't find the container with id d09e509f00abb7e14360fe755684ccccfb897dab2395141b03f921ebccd5520a
Sep 08 18:34:02.399322 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:02.399293    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4396b3d0_3459_453a_80f8_468b02bee7cb.slice/crio-3ffbfb48dd736135cf4adb487bb1acb3fcaefd8652da89953b4fe822574335a8 WatchSource:0}: Error finding container 3ffbfb48dd736135cf4adb487bb1acb3fcaefd8652da89953b4fe822574335a8: Status 404 returned error can't find the container with id 3ffbfb48dd736135cf4adb487bb1acb3fcaefd8652da89953b4fe822574335a8
Sep 08 18:34:02.407944 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:02.407915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9e740a_de51_49cf_86b7_bbbc23263d53.slice/crio-19d76b11e35dbea5b331d75597793eabaa333b04e9e4561f262e1fa44564a310 WatchSource:0}: Error finding container 19d76b11e35dbea5b331d75597793eabaa333b04e9e4561f262e1fa44564a310: Status 404 returned error can't find the container with id 19d76b11e35dbea5b331d75597793eabaa333b04e9e4561f262e1fa44564a310
Sep 08 18:34:02.648717 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.648676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:02.648918 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.648872    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:02.648992 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.648963    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:34:03.648942128 +0000 UTC m=+4.003035442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:02.750088 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.750046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:02.750279 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.750203    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:02.750348 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.750281    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret podName:e680fcd6-db0f-49ff-88c8-94c448785e49 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:03.75026223 +0000 UTC m=+4.104355548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret") pod "global-pull-secret-syncer-t5hjx" (UID: "e680fcd6-db0f-49ff-88c8-94c448785e49") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:02.850851 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:02.850807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:02.851043 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.850997    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:34:02.851043 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.851023    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:34:02.851043 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.851035    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dxttk for pod openshift-network-diagnostics/network-check-target-9b4q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:02.851223 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:02.851097    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk podName:909a461c-5e63-4736-ad00-247aa33384da nodeName:}" failed. No retries permitted until 2026-09-08 18:34:03.851079915 +0000 UTC m=+4.205173250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxttk" (UniqueName: "kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk") pod "network-check-target-9b4q4" (UID: "909a461c-5e63-4736-ad00-247aa33384da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:03.070233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.070134    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:29:02 +0000 UTC" deadline="2028-05-19 22:59:55.67245341 +0000 UTC"
Sep 08 18:34:03.070233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.070180    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14860h25m52.602284047s"
Sep 08 18:34:03.169538 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.169496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"3ffbfb48dd736135cf4adb487bb1acb3fcaefd8652da89953b4fe822574335a8"}
Sep 08 18:34:03.171060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.171030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bqqz7" event={"ID":"24f743c5-c5d6-47ce-8e49-1865602059a9","Type":"ContainerStarted","Data":"8e5480d3041f055f9ca1bf459d71ff647c41f3d0be2430ffb86f46b530a63cad"}
Sep 08 18:34:03.173124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.173089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw929" event={"ID":"f05e2e6d-161d-4996-a7c6-e060e84fd1d1","Type":"ContainerStarted","Data":"7b908a61ea5006ff2b9e43403cdfe0c767600ecb8de62ebbc954dc3e986ffef6"}
Sep 08 18:34:03.174964 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.174941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kslr8" event={"ID":"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a","Type":"ContainerStarted","Data":"bc45a413744ec9d9fa62eb9e09d05d2e58728576e63bf55cc54886843f0150f3"}
Sep 08 18:34:03.176583 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.176560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tl9d2" event={"ID":"a37d7aff-75ca-45b8-84ce-38c19e79bb69","Type":"ContainerStarted","Data":"966908ebaf3af7169593a3aaed81cb4588f417a1352821decbe677960c713284"}
Sep 08 18:34:03.179246 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.179215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zn98k" event={"ID":"b86a0f2d-bf4a-4070-9e14-00e349e64f8f","Type":"ContainerStarted","Data":"e4d7b35faa1a730839fc11c84b033fff176bfe19e15c55d835e83efbf36aa383"}
Sep 08 18:34:03.181065 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.181040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r" event={"ID":"db9e740a-de51-49cf-86b7-bbbc23263d53","Type":"ContainerStarted","Data":"19d76b11e35dbea5b331d75597793eabaa333b04e9e4561f262e1fa44564a310"}
Sep 08 18:34:03.183227 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.183200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86pss" event={"ID":"c50318ae-5b05-435f-9ba8-d2552408cdca","Type":"ContainerStarted","Data":"d09e509f00abb7e14360fe755684ccccfb897dab2395141b03f921ebccd5520a"}
Sep 08 18:34:03.184846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.184820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rn2xb" event={"ID":"eb44a0dc-533c-4750-91aa-766b956e2465","Type":"ContainerStarted","Data":"9c1f1751beb9c0efa08b14df2fc9f806803fbad8c47d072f71f5a6944e49a548"}
Sep 08 18:34:03.658236 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.658193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:03.658430 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:03.658405    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:03.658512 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:03.658482    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:34:05.658460498 +0000 UTC m=+6.012553817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:03.759619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.759575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:03.759813 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:03.759753    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:03.759876 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:03.759825    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret podName:e680fcd6-db0f-49ff-88c8-94c448785e49 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:05.759805664 +0000 UTC m=+6.113898993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret") pod "global-pull-secret-syncer-t5hjx" (UID: "e680fcd6-db0f-49ff-88c8-94c448785e49") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:03.860038 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:03.859987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:03.860309 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:03.860223    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:34:03.860309 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:03.860247    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:34:03.860309 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:03.860261    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dxttk for pod openshift-network-diagnostics/network-check-target-9b4q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:03.860577 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:03.860326    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk podName:909a461c-5e63-4736-ad00-247aa33384da nodeName:}" failed. No retries permitted until 2026-09-08 18:34:05.860305682 +0000 UTC m=+6.214399011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxttk" (UniqueName: "kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk") pod "network-check-target-9b4q4" (UID: "909a461c-5e63-4736-ad00-247aa33384da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:04.070552 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:04.070429    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:29:02 +0000 UTC" deadline="2028-05-17 10:32:43.599344568 +0000 UTC"
Sep 08 18:34:04.070552 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:04.070500    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14799h58m39.528849611s"
Sep 08 18:34:04.160284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:04.159280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:04.160284 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:04.159441    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:04.160284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:04.159908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:04.160284 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:04.160019    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:04.160284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:04.160092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:04.160284 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:04.160177    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:05.677388 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:05.677098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:05.677927 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:05.677595    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:05.677927 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:05.677717    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:34:09.67768301 +0000 UTC m=+10.031776329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:05.779311 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:05.778746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:05.779311 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:05.778899    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:05.779311 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:05.778963    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret podName:e680fcd6-db0f-49ff-88c8-94c448785e49 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:09.778945063 +0000 UTC m=+10.133038396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret") pod "global-pull-secret-syncer-t5hjx" (UID: "e680fcd6-db0f-49ff-88c8-94c448785e49") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:05.879391 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:05.879339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:05.879600 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:05.879560    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:34:05.879600 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:05.879586    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:34:05.879600 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:05.879599    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dxttk for pod openshift-network-diagnostics/network-check-target-9b4q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:05.879769 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:05.879660    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk podName:909a461c-5e63-4736-ad00-247aa33384da nodeName:}" failed. No retries permitted until 2026-09-08 18:34:09.879642297 +0000 UTC m=+10.233735625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxttk" (UniqueName: "kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk") pod "network-check-target-9b4q4" (UID: "909a461c-5e63-4736-ad00-247aa33384da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:06.157421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:06.157388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:06.157626 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:06.157395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:06.157626 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:06.157550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:06.157758 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:06.157642    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:06.157758 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:06.157696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:06.157870 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:06.157777    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:08.156958 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:08.156921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:08.157389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:08.156991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:08.157389 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:08.157107    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:08.157389 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:08.157209    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:08.157389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:08.157336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:08.157653 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:08.157611    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:09.708179 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:09.708144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:09.708602 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:09.708329    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:09.708602 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:09.708423    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:34:17.708400644 +0000 UTC m=+18.062493972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:09.808833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:09.808793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:09.809028 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:09.808929    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:09.809028 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:09.809010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret podName:e680fcd6-db0f-49ff-88c8-94c448785e49 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:17.808989499 +0000 UTC m=+18.163082829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret") pod "global-pull-secret-syncer-t5hjx" (UID: "e680fcd6-db0f-49ff-88c8-94c448785e49") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:09.909812 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:09.909723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:09.909992 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:09.909864    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:34:09.909992 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:09.909893    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:34:09.909992 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:09.909906    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dxttk for pod openshift-network-diagnostics/network-check-target-9b4q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:09.909992 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:09.909971    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk podName:909a461c-5e63-4736-ad00-247aa33384da nodeName:}" failed. No retries permitted until 2026-09-08 18:34:17.909953106 +0000 UTC m=+18.264046431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxttk" (UniqueName: "kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk") pod "network-check-target-9b4q4" (UID: "909a461c-5e63-4736-ad00-247aa33384da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:10.157893 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:10.157754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:10.157893 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:10.157855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:10.157893 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:10.157892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:10.158156 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:10.157860    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:10.158156 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:10.157995    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:10.158156 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:10.158106    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:12.157400 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:12.157361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:12.157833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:12.157366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:12.157833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:12.157505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:12.157833 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:12.157500    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:12.157833 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:12.157609    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:12.157833 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:12.157703    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:14.156765 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:14.156730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:14.157187 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:14.156730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:14.157187 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:14.156855    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:14.157187 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:14.156949    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:14.157187 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:14.156732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:14.157187 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:14.157037    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:16.157039 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:16.157001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:16.157489 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:16.157010    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:16.157489 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:16.157149    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:16.157489 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:16.157188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:16.157489 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:16.157312    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:16.157489 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:16.157477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:17.769947 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:17.769901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:17.770576 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:17.770076    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:17.770576 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:17.770146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:34:33.770125387 +0000 UTC m=+34.124218721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:17.870519 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:17.870480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:17.870675 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:17.870608    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:17.870675 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:17.870667    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret podName:e680fcd6-db0f-49ff-88c8-94c448785e49 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:33.87065292 +0000 UTC m=+34.224746232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret") pod "global-pull-secret-syncer-t5hjx" (UID: "e680fcd6-db0f-49ff-88c8-94c448785e49") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:17.971812 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:17.971776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:17.971989 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:17.971938    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:34:17.971989 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:17.971955    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:34:17.971989 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:17.971965    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dxttk for pod openshift-network-diagnostics/network-check-target-9b4q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:17.972119 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:17.972016    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk podName:909a461c-5e63-4736-ad00-247aa33384da nodeName:}" failed. No retries permitted until 2026-09-08 18:34:33.972002334 +0000 UTC m=+34.326095665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxttk" (UniqueName: "kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk") pod "network-check-target-9b4q4" (UID: "909a461c-5e63-4736-ad00-247aa33384da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:18.156610 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:18.156576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:18.156792 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:18.156576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:18.156792 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:18.156724    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:18.156792 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:18.156592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:18.156962 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:18.156795    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:18.156962 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:18.156902    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:20.159814 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:20.158477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:20.159814 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:20.158610    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:20.159814 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:20.159073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:20.159814 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:20.159207    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:20.159814 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:20.159270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:20.159814 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:20.159426    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:21.232940 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.232608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal" event={"ID":"e01954d7983c7d343489f8e29f686ea0","Type":"ContainerStarted","Data":"5ecdcd61e1d280d3514d6d455104a0bc590e618a55b2a71d4d6ea8379de4f22d"}
Sep 08 18:34:21.235407 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.235387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:34:21.235717 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.235699    2579 generic.go:356] "Generic (PLEG): container finished" podID="4396b3d0-3459-453a-80f8-468b02bee7cb" containerID="99e15245b2269beb0a079d7ad74e0e7aadc9acec5d3dec31a2eb3293d9c6000c" exitCode=1
Sep 08 18:34:21.235790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.235771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"ee3cc83288236d2c7836d92c4d7a5c65e4ba05b2b56f39f95062e9371ac83692"}
Sep 08 18:34:21.235845 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.235830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"cb21a53da6baa3dc155f4c11584c43b790546b0a9cc1b39a90f82b15b047a4ab"}
Sep 08 18:34:21.235894 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.235874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"f73057263fcfe4c3c7ba15a26b834d546b9194d3e27689e605ee394c6ea72ca2"}
Sep 08 18:34:21.235932 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.235906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"a2d92881687637f7d8073d8db496308a16a1d5004f1254840967bccb69de581b"}
Sep 08 18:34:21.235979 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.235937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerDied","Data":"99e15245b2269beb0a079d7ad74e0e7aadc9acec5d3dec31a2eb3293d9c6000c"}
Sep 08 18:34:21.235979 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.235975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"7579daa730257b0dc7ca40bbb46490e797cb74d078042a05ee1ea3702635cae4"}
Sep 08 18:34:21.237170 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.237141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bqqz7" event={"ID":"24f743c5-c5d6-47ce-8e49-1865602059a9","Type":"ContainerStarted","Data":"0c6ccc0e291c1d7a8ccfef680772d1025755f5aa954a3f5ae7290f58354b4675"}
Sep 08 18:34:21.238537 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.238519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw929" event={"ID":"f05e2e6d-161d-4996-a7c6-e060e84fd1d1","Type":"ContainerStarted","Data":"8af6aeb182b3fad4f1056297e51a16519748dcb92bf2a1a3e8f2e6b258bede36"}
Sep 08 18:34:21.239837 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.239820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kslr8" event={"ID":"21b4eada-cc2e-4655-90b8-38e6f9bbcf6a","Type":"ContainerStarted","Data":"36722253baf4ba0e43b43e04cdbf316565990b6fc9fa8bd8b247332e20a3f5e8"}
Sep 08 18:34:21.241105 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.241085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tl9d2" event={"ID":"a37d7aff-75ca-45b8-84ce-38c19e79bb69","Type":"ContainerStarted","Data":"c1b057b0e9db681cc0d677cabbc90bb22880122275a5f0440b3e748dc394a75a"}
Sep 08 18:34:21.242483 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.242464    2579 generic.go:356] "Generic (PLEG): container finished" podID="9036dac6183b2c3d77595670f1fe6a01" containerID="44423789bb009b313f2fb680f3571440c0440ca5055cc4bd016478e9e76c8e52" exitCode=0
Sep 08 18:34:21.242545 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.242523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal" event={"ID":"9036dac6183b2c3d77595670f1fe6a01","Type":"ContainerDied","Data":"44423789bb009b313f2fb680f3571440c0440ca5055cc4bd016478e9e76c8e52"}
Sep 08 18:34:21.244042 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.244002    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-26.ec2.internal" podStartSLOduration=19.243989079 podStartE2EDuration="19.243989079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:21.243712631 +0000 UTC m=+21.597805967" watchObservedRunningTime="2026-09-08 18:34:21.243989079 +0000 UTC m=+21.598082412"
Sep 08 18:34:21.244146 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.244083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r" event={"ID":"db9e740a-de51-49cf-86b7-bbbc23263d53","Type":"ContainerStarted","Data":"b61df301605f93455401cdb6a7cc3ce5d9bea60b4f9fb8b7fdfcac744aab74b0"}
Sep 08 18:34:21.245623 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.245603    2579 generic.go:356] "Generic (PLEG): container finished" podID="c50318ae-5b05-435f-9ba8-d2552408cdca" containerID="086a3520005c4745fd99e43d763d26c56e839fc158c6183d4dbf00ae94e643e2" exitCode=0
Sep 08 18:34:21.245719 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.245666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86pss" event={"ID":"c50318ae-5b05-435f-9ba8-d2552408cdca","Type":"ContainerDied","Data":"086a3520005c4745fd99e43d763d26c56e839fc158c6183d4dbf00ae94e643e2"}
Sep 08 18:34:21.247137 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.247113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rn2xb" event={"ID":"eb44a0dc-533c-4750-91aa-766b956e2465","Type":"ContainerStarted","Data":"944bbbf7da16b8f6e588b4598c6a019957a1c5e7ec09d897a4df934f51347bc2"}
Sep 08 18:34:21.258432 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.258391    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kslr8" podStartSLOduration=3.533846194 podStartE2EDuration="21.258331484s" totalImagesPullingTime="17.72448529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:21.258194678 +0000 UTC m=+21.612288012" watchObservedRunningTime="2026-09-08 18:34:21.258331484 +0000 UTC m=+21.612424851"
Sep 08 18:34:21.270172 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.270124    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bqqz7" podStartSLOduration=3.563708765 podStartE2EDuration="21.270107857s" totalImagesPullingTime="17.706399092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:21.269477267 +0000 UTC m=+21.623570599" watchObservedRunningTime="2026-09-08 18:34:21.270107857 +0000 UTC m=+21.624201190"
Sep 08 18:34:21.296328 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.296267    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tl9d2" podStartSLOduration=3.54025537 podStartE2EDuration="21.296247937s" totalImagesPullingTime="17.755992567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:21.29614158 +0000 UTC m=+21.650234913" watchObservedRunningTime="2026-09-08 18:34:21.296247937 +0000 UTC m=+21.650341273"
Sep 08 18:34:21.310166 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.310128    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bw929" podStartSLOduration=3.121545099 podStartE2EDuration="21.310113982s" totalImagesPullingTime="18.188568883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:21.309820076 +0000 UTC m=+21.663913408" watchObservedRunningTime="2026-09-08 18:34:21.310113982 +0000 UTC m=+21.664207314"
Sep 08 18:34:21.369027 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.368972    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rn2xb" podStartSLOduration=7.443367416 podStartE2EDuration="21.36895659s" totalImagesPullingTime="13.925589174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:21.325902151 +0000 UTC m=+21.679995503" watchObservedRunningTime="2026-09-08 18:34:21.36895659 +0000 UTC m=+21.723049923"
Sep 08 18:34:21.828153 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:21.828135    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 18:34:22.090071 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.089944    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T18:34:21.828150303Z","UUID":"d9814dbe-1da8-4b98-b773-6e95e3dcbeef","Handler":null,"Name":"","Endpoint":""}
Sep 08 18:34:22.091515 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.091498    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 18:34:22.091515 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.091526    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 18:34:22.156926 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.156864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:22.157087 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.156865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:22.157087 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.156864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:22.157087 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:22.157036    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:22.157243 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:22.157171    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:22.157290 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:22.157257    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:22.250888 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.250847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zn98k" event={"ID":"b86a0f2d-bf4a-4070-9e14-00e349e64f8f","Type":"ContainerStarted","Data":"cdf84d9bc96a60ceaf9eb7a027f3d13e676ef8f1dbced525068a06f8feac385a"}
Sep 08 18:34:22.252700 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.252677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal" event={"ID":"9036dac6183b2c3d77595670f1fe6a01","Type":"ContainerStarted","Data":"9abc9f8975624d18808aea4dd7b8c7dde8b2acf8f571385e546743dd2d75559b"}
Sep 08 18:34:22.254562 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.254539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r" event={"ID":"db9e740a-de51-49cf-86b7-bbbc23263d53","Type":"ContainerStarted","Data":"5fb8c07b775f21ca23a26cf674f84236cbd2525445af06f1e2474036ee1fde2a"}
Sep 08 18:34:22.262625 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.262588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zn98k" podStartSLOduration=4.512839027 podStartE2EDuration="22.262577292s" totalImagesPullingTime="17.749738265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:22.262506112 +0000 UTC m=+22.616599460" watchObservedRunningTime="2026-09-08 18:34:22.262577292 +0000 UTC m=+22.616670622"
Sep 08 18:34:22.555436 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.555336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:22.556101 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.556083    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:22.569208 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:22.569163    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-26.ec2.internal" podStartSLOduration=20.569147412 podStartE2EDuration="20.569147412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:22.27365982 +0000 UTC m=+22.627753152" watchObservedRunningTime="2026-09-08 18:34:22.569147412 +0000 UTC m=+22.923240745"
Sep 08 18:34:23.260110 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:23.260026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r" event={"ID":"db9e740a-de51-49cf-86b7-bbbc23263d53","Type":"ContainerStarted","Data":"e45efe11ba39800173bff22da2a4165bd19b385cdd74407d87a0fbf9b307a78d"}
Sep 08 18:34:23.263732 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:23.263706    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:34:23.264216 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:23.264191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"4c7d9f95e1d166f3d985f87a0ec05cd6d24312b64844e8b7648c36e0968773c8"}
Sep 08 18:34:23.276222 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:23.276181    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h865r" podStartSLOduration=2.638660087 podStartE2EDuration="22.276165495s" totalImagesPullingTime="19.637505408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:23.27556379 +0000 UTC m=+23.629657122" watchObservedRunningTime="2026-09-08 18:34:23.276165495 +0000 UTC m=+23.630258851"
Sep 08 18:34:24.157294 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:24.157117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:24.157486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:24.157119    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:24.157486 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:24.157404    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:24.157486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:24.157181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:24.157486 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:24.157447    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:24.157644 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:24.157600    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:24.266227 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:24.266196    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 18:34:25.272809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:25.272780    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:34:25.273839 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:25.273479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"d1a040f095a03491b4546f1a245a62541b41a57533718c649da4cf59bae2003d"}
Sep 08 18:34:25.273969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:25.273903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:25.274199 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:25.274176    2579 scope.go:117] "RemoveContainer" containerID="99e15245b2269beb0a079d7ad74e0e7aadc9acec5d3dec31a2eb3293d9c6000c"
Sep 08 18:34:25.293435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:25.293405    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:26.156958 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:26.156731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:26.157238 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:26.157206    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:26.157402 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:26.156731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:26.157402 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:26.156734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:26.157520 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:26.157420    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:26.157576 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:26.157547    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:27.022876 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.022839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5hjx"]
Sep 08 18:34:27.023449 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.022973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:27.023449 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:27.023079    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:27.023627 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.023609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9b4q4"]
Sep 08 18:34:27.023703 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.023692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:27.023780 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:27.023765    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:27.026142 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.026120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8mlg8"]
Sep 08 18:34:27.026260 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.026209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:27.026322 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:27.026282    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:27.282321 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.282289    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:34:27.282740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.282703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" event={"ID":"4396b3d0-3459-453a-80f8-468b02bee7cb","Type":"ContainerStarted","Data":"2f706f4141ff15850e12f9b3c4a070dfa81a74472348241aa00c8ef182362d68"}
Sep 08 18:34:27.282894 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.282881    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:27.282970 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.282958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:27.284750 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.284722    2579 generic.go:356] "Generic (PLEG): container finished" podID="c50318ae-5b05-435f-9ba8-d2552408cdca" containerID="417b310cb95b10cbc5d6e0ed2efc46d3af40a0c92b40951149f1edf032a4cad0" exitCode=0
Sep 08 18:34:27.284843 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.284760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86pss" event={"ID":"c50318ae-5b05-435f-9ba8-d2552408cdca","Type":"ContainerDied","Data":"417b310cb95b10cbc5d6e0ed2efc46d3af40a0c92b40951149f1edf032a4cad0"}
Sep 08 18:34:27.297918 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.297896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:27.306501 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:27.306459    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp" podStartSLOduration=8.560165534 podStartE2EDuration="26.306443339s" totalImagesPullingTime="17.746277805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:27.305676433 +0000 UTC m=+27.659769792" watchObservedRunningTime="2026-09-08 18:34:27.306443339 +0000 UTC m=+27.660536672"
Sep 08 18:34:28.891522 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:28.891485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:28.892154 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:28.891623    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 18:34:28.892154 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:28.892054    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bqqz7"
Sep 08 18:34:29.157084 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:29.157012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:29.157222 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:29.157014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:29.157222 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:29.157114    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:29.157222 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:29.157196    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:29.157222 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:29.157025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:29.157356 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:29.157275    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:29.291618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:29.291583    2579 generic.go:356] "Generic (PLEG): container finished" podID="c50318ae-5b05-435f-9ba8-d2552408cdca" containerID="ae8606a4ab3d6b4c6cefbf1c660500899058347f1e17e93addbca7f7927bd397" exitCode=0
Sep 08 18:34:29.291758 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:29.291656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86pss" event={"ID":"c50318ae-5b05-435f-9ba8-d2552408cdca","Type":"ContainerDied","Data":"ae8606a4ab3d6b4c6cefbf1c660500899058347f1e17e93addbca7f7927bd397"}
Sep 08 18:34:30.296486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:30.296235    2579 generic.go:356] "Generic (PLEG): container finished" podID="c50318ae-5b05-435f-9ba8-d2552408cdca" containerID="7d2abbb5f16ab2c5a2e0b36d5c8ddf9c31504300761604686baf7b103780e9c6" exitCode=0
Sep 08 18:34:30.296486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:30.296317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86pss" event={"ID":"c50318ae-5b05-435f-9ba8-d2552408cdca","Type":"ContainerDied","Data":"7d2abbb5f16ab2c5a2e0b36d5c8ddf9c31504300761604686baf7b103780e9c6"}
Sep 08 18:34:31.156870 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:31.156830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:31.157055 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:31.156884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:31.157055 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:31.156850    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:31.157055 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:31.156948    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:31.157055 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:31.156999    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:31.157183 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:31.157083    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:33.157162 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.157113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:33.157585 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.157113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:33.157585 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.157243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:33.157585 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.157240    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5hjx" podUID="e680fcd6-db0f-49ff-88c8-94c448785e49"
Sep 08 18:34:33.157585 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.157332    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b4q4" podUID="909a461c-5e63-4736-ad00-247aa33384da"
Sep 08 18:34:33.157585 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.157413    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:34:33.444799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.444773    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-26.ec2.internal" event="NodeReady"
Sep 08 18:34:33.444960 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.444883    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 18:34:33.474925 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.474892    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x6cmn"]
Sep 08 18:34:33.476767 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.476749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:33.478265 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.478243    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64887f6446-mzzw4"]
Sep 08 18:34:33.479018 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.478992    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xjn8s\""
Sep 08 18:34:33.479113 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.479088    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 18:34:33.479312 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.479298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 18:34:33.480089 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.480071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.482048 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.481875    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 18:34:33.482443 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.482420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xlp48\""
Sep 08 18:34:33.482561 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.482445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 18:34:33.482561 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.482465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 18:34:33.486488 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.486470    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 18:34:33.489630 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.489277    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x6cmn"]
Sep 08 18:34:33.490100 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.490080    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-flcd8"]
Sep 08 18:34:33.492051 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.492031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-flcd8"
Sep 08 18:34:33.492394 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.492372    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nqt6r"]
Sep 08 18:34:33.494389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.494165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 18:34:33.494389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.494211    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 18:34:33.494556 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.494542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9b5jk\""
Sep 08 18:34:33.495718 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.494885    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 18:34:33.496141 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.496120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64887f6446-mzzw4"]
Sep 08 18:34:33.496230 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.496217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.497986 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.497754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rsh8d\""
Sep 08 18:34:33.497986 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.497801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 18:34:33.498131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.498038    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 18:34:33.500086 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.500061    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-flcd8"]
Sep 08 18:34:33.504418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.504242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nqt6r"]
Sep 08 18:34:33.581780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2f54353-feeb-4c43-8b29-9c863b8dd040-cert\") pod \"ingress-canary-flcd8\" (UID: \"e2f54353-feeb-4c43-8b29-9c863b8dd040\") " pod="openshift-ingress-canary/ingress-canary-flcd8"
Sep 08 18:34:33.581780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsr6h\" (UniqueName: \"kubernetes.io/projected/e2f54353-feeb-4c43-8b29-9c863b8dd040-kube-api-access-rsr6h\") pod \"ingress-canary-flcd8\" (UID: \"e2f54353-feeb-4c43-8b29-9c863b8dd040\") " pod="openshift-ingress-canary/ingress-canary-flcd8"
Sep 08 18:34:33.581949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.581949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ct9q\" (UniqueName: \"kubernetes.io/projected/c1654ffb-c4d0-421b-b162-28c6f1377f68-kube-api-access-8ct9q\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.581949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-installation-pull-secrets\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.581949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1654ffb-c4d0-421b-b162-28c6f1377f68-config-volume\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.581949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.582115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-image-registry-private-configuration\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.582115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.581995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-ca-trust-extracted\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.582115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.582021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-bound-sa-token\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.582115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.582053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:33.582115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.582074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswzv\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-kube-api-access-fswzv\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.582115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.582105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1654ffb-c4d0-421b-b162-28c6f1377f68-tmp-dir\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.582283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.582120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-trusted-ca\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.582283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.582135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0942923-36ad-4be6-afe4-b5c44d783a1c-nginx-conf\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:33.582283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.582150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-certificates\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.683113 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.683281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-image-registry-private-configuration\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.683281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-ca-trust-extracted\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.683281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-bound-sa-token\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.683281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:33.683281 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.683236    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:34:33.683281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswzv\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-kube-api-access-fswzv\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.683281 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.683253    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.683307    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:34.183293181 +0000 UTC m=+34.537386510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1654ffb-c4d0-421b-b162-28c6f1377f68-tmp-dir\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-trusted-ca\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0942923-36ad-4be6-afe4-b5c44d783a1c-nginx-conf\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.683397    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-certificates\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.683449    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:34.183436047 +0000 UTC m=+34.537529360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2f54353-feeb-4c43-8b29-9c863b8dd040-cert\") pod \"ingress-canary-flcd8\" (UID: \"e2f54353-feeb-4c43-8b29-9c863b8dd040\") " pod="openshift-ingress-canary/ingress-canary-flcd8"
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsr6h\" (UniqueName: \"kubernetes.io/projected/e2f54353-feeb-4c43-8b29-9c863b8dd040-kube-api-access-rsr6h\") pod \"ingress-canary-flcd8\" (UID: \"e2f54353-feeb-4c43-8b29-9c863b8dd040\") " pod="openshift-ingress-canary/ingress-canary-flcd8"
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ct9q\" (UniqueName: \"kubernetes.io/projected/c1654ffb-c4d0-421b-b162-28c6f1377f68-kube-api-access-8ct9q\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.683624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-installation-pull-secrets\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.684116 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1654ffb-c4d0-421b-b162-28c6f1377f68-config-volume\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.684116 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-ca-trust-extracted\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.684116 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1654ffb-c4d0-421b-b162-28c6f1377f68-tmp-dir\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.684116 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.683914    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:34:33.684116 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.683971    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:34.183953007 +0000 UTC m=+34.538046325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:34:33.684116 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.683986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-certificates\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.684408 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.684208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1654ffb-c4d0-421b-b162-28c6f1377f68-config-volume\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.684408 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.684299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0942923-36ad-4be6-afe4-b5c44d783a1c-nginx-conf\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:33.684408 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.684382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-trusted-ca\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.687473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.687450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-installation-pull-secrets\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.687473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.687460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2f54353-feeb-4c43-8b29-9c863b8dd040-cert\") pod \"ingress-canary-flcd8\" (UID: \"e2f54353-feeb-4c43-8b29-9c863b8dd040\") " pod="openshift-ingress-canary/ingress-canary-flcd8"
Sep 08 18:34:33.687658 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.687501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-image-registry-private-configuration\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.691195 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.691170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-bound-sa-token\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.691566 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.691537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswzv\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-kube-api-access-fswzv\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:33.691817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.691801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsr6h\" (UniqueName: \"kubernetes.io/projected/e2f54353-feeb-4c43-8b29-9c863b8dd040-kube-api-access-rsr6h\") pod \"ingress-canary-flcd8\" (UID: \"e2f54353-feeb-4c43-8b29-9c863b8dd040\") " pod="openshift-ingress-canary/ingress-canary-flcd8"
Sep 08 18:34:33.692029 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.692010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ct9q\" (UniqueName: \"kubernetes.io/projected/c1654ffb-c4d0-421b-b162-28c6f1377f68-kube-api-access-8ct9q\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:33.784530 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.784438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:33.784692 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.784607    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:33.784692 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.784675    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:35:05.784658517 +0000 UTC m=+66.138751828 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:34:33.808249 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.808204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-flcd8"
Sep 08 18:34:33.886168 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.885924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:33.886405 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.886077    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:33.886405 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.886278    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret podName:e680fcd6-db0f-49ff-88c8-94c448785e49 nodeName:}" failed. No retries permitted until 2026-09-08 18:35:05.88626321 +0000 UTC m=+66.240356526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret") pod "global-pull-secret-syncer-t5hjx" (UID: "e680fcd6-db0f-49ff-88c8-94c448785e49") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:34:33.987007 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:33.986971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:33.987186 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.987154    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:34:33.987186 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.987182    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:34:33.987253 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.987195    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dxttk for pod openshift-network-diagnostics/network-check-target-9b4q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:33.987253 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:33.987241    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk podName:909a461c-5e63-4736-ad00-247aa33384da nodeName:}" failed. No retries permitted until 2026-09-08 18:35:05.987228692 +0000 UTC m=+66.341322002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxttk" (UniqueName: "kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk") pod "network-check-target-9b4q4" (UID: "909a461c-5e63-4736-ad00-247aa33384da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:34:34.086574 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:34.086541    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-flcd8"]
Sep 08 18:34:34.188329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:34.188285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:34.188370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:34.188433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:34.188500    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:34.188525    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:34.188547    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:34.188509    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:34.188593    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:35.188571026 +0000 UTC m=+35.542664344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:34.188669    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:35.188653798 +0000 UTC m=+35.542747120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:34:34.188798 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:34.188683    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:35.188674642 +0000 UTC m=+35.542767953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:34:35.157190 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.157153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:34:35.157190 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.157185    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:34:35.157461 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.157156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:34:35.159401 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.159366    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 18:34:35.159401 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.159382    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 18:34:35.159591 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.159505    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 18:34:35.160034 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.159912    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmn47\""
Sep 08 18:34:35.160155 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.160036    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z79bg\""
Sep 08 18:34:35.160155 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.160079    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 18:34:35.198152 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.198119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.198187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:35.198249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:35.198291    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:35.198391    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:35.198409    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:37.198385848 +0000 UTC m=+37.552479182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:35.198453    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:37.198435248 +0000 UTC m=+37.552528577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:35.198392    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:35.198476    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:34:35.198578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:35.198511    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:37.198502217 +0000 UTC m=+37.552595528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:34:35.334562 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:34:35.334525    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f54353_feeb_4c43_8b29_9c863b8dd040.slice/crio-4a940f531ab52984df99b6bc4a6c2d63f5362480eaa66b03d3a2f5708c761a88 WatchSource:0}: Error finding container 4a940f531ab52984df99b6bc4a6c2d63f5362480eaa66b03d3a2f5708c761a88: Status 404 returned error can't find the container with id 4a940f531ab52984df99b6bc4a6c2d63f5362480eaa66b03d3a2f5708c761a88
Sep 08 18:34:36.315111 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:36.315077    2579 generic.go:356] "Generic (PLEG): container finished" podID="c50318ae-5b05-435f-9ba8-d2552408cdca" containerID="aec347ba11d61b7efc76ccd1b85f80c8200ad8928f1fe20edcd707107c996eb1" exitCode=0
Sep 08 18:34:36.315776 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:36.315175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86pss" event={"ID":"c50318ae-5b05-435f-9ba8-d2552408cdca","Type":"ContainerDied","Data":"aec347ba11d61b7efc76ccd1b85f80c8200ad8928f1fe20edcd707107c996eb1"}
Sep 08 18:34:36.316749 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:36.316719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-flcd8" event={"ID":"e2f54353-feeb-4c43-8b29-9c863b8dd040","Type":"ContainerStarted","Data":"4a940f531ab52984df99b6bc4a6c2d63f5362480eaa66b03d3a2f5708c761a88"}
Sep 08 18:34:37.213211 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:37.213171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:37.213235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:37.213286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:37.213322    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:37.213404    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:41.213388978 +0000 UTC m=+41.567482295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:37.213410    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:37.213427    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:37.213413    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:37.213467    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:41.21345577 +0000 UTC m=+41.567549082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:34:37.213537 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:37.213502    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:41.213488835 +0000 UTC m=+41.567582159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:34:37.323063 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:37.323005    2579 generic.go:356] "Generic (PLEG): container finished" podID="c50318ae-5b05-435f-9ba8-d2552408cdca" containerID="296711953f829052363dd188b85da08eec4a808b65a162c59b9671a9cfaaa2a8" exitCode=0
Sep 08 18:34:37.323063 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:37.323053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86pss" event={"ID":"c50318ae-5b05-435f-9ba8-d2552408cdca","Type":"ContainerDied","Data":"296711953f829052363dd188b85da08eec4a808b65a162c59b9671a9cfaaa2a8"}
Sep 08 18:34:38.328355 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:38.328300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86pss" event={"ID":"c50318ae-5b05-435f-9ba8-d2552408cdca","Type":"ContainerStarted","Data":"1a924cd280864c4cd90867b35a94830546f45c96b52a31072efa57aa20a1adf8"}
Sep 08 18:34:38.349021 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:38.348976    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-86pss" podStartSLOduration=7.734698481 podStartE2EDuration="38.34896111s" totalImagesPullingTime="30.614262629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:38.347782696 +0000 UTC m=+38.701876026" watchObservedRunningTime="2026-09-08 18:34:38.34896111 +0000 UTC m=+38.703054444"
Sep 08 18:34:38.381937 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:38.381902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:38.458564 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:38.458525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:38.548060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:38.547870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:38.664526 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:38.664488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:38.924135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:38.924010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:38.976250 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:38.976218    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:39.332435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:39.332403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-flcd8" event={"ID":"e2f54353-feeb-4c43-8b29-9c863b8dd040","Type":"ContainerStarted","Data":"6245d6d828329b7a81ac6ae10eb43c9bd5970f499d9c54fb4ebf71660c47ed37"}
Sep 08 18:34:39.351540 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:39.351496    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-flcd8" podStartSLOduration=3.223103031 podStartE2EDuration="6.351480613s" totalImagesPullingTime="3.128377582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:39.350771537 +0000 UTC m=+39.704864869" watchObservedRunningTime="2026-09-08 18:34:39.351480613 +0000 UTC m=+39.705573946"
Sep 08 18:34:40.990821 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:40.990786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:41.041802 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:41.041768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:41.245120 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:41.245032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:41.245120 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:41.245086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:41.245379 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:41.245128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:41.245379 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:41.245224    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:34:41.245379 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:41.245228    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:34:41.245379 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:41.245248    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:34:41.245379 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:41.245262    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:34:41.245379 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:41.245276    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:49.245262408 +0000 UTC m=+49.599355733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:34:41.245379 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:41.245295    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:49.245281788 +0000 UTC m=+49.599375102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:34:41.245379 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:41.245317    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:49.245301846 +0000 UTC m=+49.599395175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:34:41.642736 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:41.642706    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:46.018088 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:46.018037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:46.118921 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:46.118882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:47.008916 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:47.008879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:47.109018 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:47.108992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:47.951012 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:47.950979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:49.307312 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:49.307275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:49.307356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:49.307412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:49.307532    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:49.307560    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:49.307588    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:49.307534    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:49.307681    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:35:05.307662085 +0000 UTC m=+65.661755420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:49.307700    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:35:05.307690422 +0000 UTC m=+65.661783734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:34:49.307799 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:34:49.307714    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:05.307707359 +0000 UTC m=+65.661800672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:34:59.302679 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:59.302650    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6qmp"
Sep 08 18:34:59.451897 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:59.451850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:34:59.500899 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:34:59.500849    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:01.335703 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:01.335653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:01.386454 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:01.386415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:03.697500 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:03.697461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:03.753242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:03.753214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:05.106403 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.106330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:05.206063 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.206017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:05.318160 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.318124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:35:05.318362 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.318186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:35:05.318362 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.318240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:35:05.318362 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.318266    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:35:05.318362 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.318285    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:35:05.318362 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.318333    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:35:05.318362 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.318336    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:35:05.318595 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.318354    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:35:37.318324309 +0000 UTC m=+97.672417664 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:35:05.318595 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.318409    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:37.318392411 +0000 UTC m=+97.672485721 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:35:05.318595 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.318424    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:35:37.318414848 +0000 UTC m=+97.672508160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:35:05.491397 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.491303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:05.821972 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.821879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:35:05.824038 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.824019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 18:35:05.832659 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.832626    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 18:35:05.832724 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:05.832706    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:36:09.832690568 +0000 UTC m=+130.186783892 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : secret "metrics-daemon-secret" not found
Sep 08 18:35:05.922257 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.922221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:35:05.924154 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.924137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 18:35:05.935833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:05.935799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e680fcd6-db0f-49ff-88c8-94c448785e49-original-pull-secret\") pod \"global-pull-secret-syncer-t5hjx\" (UID: \"e680fcd6-db0f-49ff-88c8-94c448785e49\") " pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:35:06.023335 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.023299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:35:06.025259 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.025242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 18:35:06.035174 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.035155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 18:35:06.046838 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.046817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxttk\" (UniqueName: \"kubernetes.io/projected/909a461c-5e63-4736-ad00-247aa33384da-kube-api-access-dxttk\") pod \"network-check-target-9b4q4\" (UID: \"909a461c-5e63-4736-ad00-247aa33384da\") " pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:35:06.076973 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.076901    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5hjx"
Sep 08 18:35:06.083827 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.083807    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmn47\""
Sep 08 18:35:06.092427 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.092396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:35:06.345974 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.345942    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5hjx"]
Sep 08 18:35:06.349235 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:35:06.349205    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode680fcd6_db0f_49ff_88c8_94c448785e49.slice/crio-2c2afcc69f688789ca46fe8016df2bcb6251a2667b9c64091d756fdc4a9e7040 WatchSource:0}: Error finding container 2c2afcc69f688789ca46fe8016df2bcb6251a2667b9c64091d756fdc4a9e7040: Status 404 returned error can't find the container with id 2c2afcc69f688789ca46fe8016df2bcb6251a2667b9c64091d756fdc4a9e7040
Sep 08 18:35:06.375035 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.375009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9b4q4"]
Sep 08 18:35:06.378653 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:35:06.378625    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909a461c_5e63_4736_ad00_247aa33384da.slice/crio-883849f3007ffec671234ad3f02bd3966a79a034122fcfdc638ec562c5b3925d WatchSource:0}: Error finding container 883849f3007ffec671234ad3f02bd3966a79a034122fcfdc638ec562c5b3925d: Status 404 returned error can't find the container with id 883849f3007ffec671234ad3f02bd3966a79a034122fcfdc638ec562c5b3925d
Sep 08 18:35:06.395229 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.395201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9b4q4" event={"ID":"909a461c-5e63-4736-ad00-247aa33384da","Type":"ContainerStarted","Data":"883849f3007ffec671234ad3f02bd3966a79a034122fcfdc638ec562c5b3925d"}
Sep 08 18:35:06.396253 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:06.396232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5hjx" event={"ID":"e680fcd6-db0f-49ff-88c8-94c448785e49","Type":"ContainerStarted","Data":"2c2afcc69f688789ca46fe8016df2bcb6251a2667b9c64091d756fdc4a9e7040"}
Sep 08 18:35:12.413925 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:12.413891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9b4q4" event={"ID":"909a461c-5e63-4736-ad00-247aa33384da","Type":"ContainerStarted","Data":"923628e930370f2eaff7916e7c00fa5c5d28cf071999564c6a0537a883d68357"}
Sep 08 18:35:12.414464 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:12.413995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:35:12.415354 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:12.415317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5hjx" event={"ID":"e680fcd6-db0f-49ff-88c8-94c448785e49","Type":"ContainerStarted","Data":"4b9e29cc4b2f4bd228902a86f31676d1d8dccda1c01b39e414654c7d81a3d3ea"}
Sep 08 18:35:12.431368 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:12.431310    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9b4q4" podStartSLOduration=66.862162589 podStartE2EDuration="1m12.431297382s" totalImagesPullingTime="5.569134793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:12.430310853 +0000 UTC m=+72.784404185" watchObservedRunningTime="2026-09-08 18:35:12.431297382 +0000 UTC m=+72.785390706"
Sep 08 18:35:12.442806 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:12.442768    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t5hjx" podStartSLOduration=65.852359066 podStartE2EDuration="1m11.442753799s" totalImagesPullingTime="5.590394733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:12.442139498 +0000 UTC m=+72.796232831" watchObservedRunningTime="2026-09-08 18:35:12.442753799 +0000 UTC m=+72.796847132"
Sep 08 18:35:26.045695 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:26.045587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:26.105192 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:26.105160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:35:37.344065 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:37.344027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:37.344081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:37.344174    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:37.344196    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:37.344219    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:36:41.344207053 +0000 UTC m=+161.698300363 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:37.344212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:37.344276    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:37.344295    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:37.344277    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:36:41.344258784 +0000 UTC m=+161.698352101 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:35:37.344564 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:35:37.344361    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:36:41.344331184 +0000 UTC m=+161.698424511 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:35:43.420385 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:43.420328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9b4q4"
Sep 08 18:35:55.989682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:55.989648    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"]
Sep 08 18:35:55.991368 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:55.991331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:55.994192 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:55.994164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 18:35:55.994192 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:55.994187    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 18:35:55.994376 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:55.994203    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 18:35:55.994376 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:55.994245    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 18:35:56.002565 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.002544    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"]
Sep 08 18:35:56.073451 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.073420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47fcba70-51a4-4449-a367-35589af8f26c-tmp\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.073451 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.073449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/47fcba70-51a4-4449-a367-35589af8f26c-klusterlet-config\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.073633 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.073481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nms9w\" (UniqueName: \"kubernetes.io/projected/47fcba70-51a4-4449-a367-35589af8f26c-kube-api-access-nms9w\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.174329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.174304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nms9w\" (UniqueName: \"kubernetes.io/projected/47fcba70-51a4-4449-a367-35589af8f26c-kube-api-access-nms9w\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.174468 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.174389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47fcba70-51a4-4449-a367-35589af8f26c-tmp\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.174468 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.174412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/47fcba70-51a4-4449-a367-35589af8f26c-klusterlet-config\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.174734 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.174715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47fcba70-51a4-4449-a367-35589af8f26c-tmp\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.176776 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.176749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/47fcba70-51a4-4449-a367-35589af8f26c-klusterlet-config\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.180461 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.180441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nms9w\" (UniqueName: \"kubernetes.io/projected/47fcba70-51a4-4449-a367-35589af8f26c-kube-api-access-nms9w\") pod \"klusterlet-addon-workmgr-685868d9f8-sjk6z\" (UID: \"47fcba70-51a4-4449-a367-35589af8f26c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.300725 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.300636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:35:56.564112 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:56.564032    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"]
Sep 08 18:35:56.567120 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:35:56.567093    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fcba70_51a4_4449_a367_35589af8f26c.slice/crio-8bfeaf117bf056d16e5cb39d155d100c94633f88dce2187aaeda3a493694318f WatchSource:0}: Error finding container 8bfeaf117bf056d16e5cb39d155d100c94633f88dce2187aaeda3a493694318f: Status 404 returned error can't find the container with id 8bfeaf117bf056d16e5cb39d155d100c94633f88dce2187aaeda3a493694318f
Sep 08 18:35:57.523464 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:35:57.523429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z" event={"ID":"47fcba70-51a4-4449-a367-35589af8f26c","Type":"ContainerStarted","Data":"8bfeaf117bf056d16e5cb39d155d100c94633f88dce2187aaeda3a493694318f"}
Sep 08 18:36:00.532217 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:00.532180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z" event={"ID":"47fcba70-51a4-4449-a367-35589af8f26c","Type":"ContainerStarted","Data":"da90a62858efe042362a3db9a674f6573b635b947c91109a3523acd7b3d06816"}
Sep 08 18:36:00.532619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:00.532476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:36:00.533653 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:00.533635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:36:00.545504 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:00.545460    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z" podStartSLOduration=1.695421673 podStartE2EDuration="5.545446771s" totalImagesPullingTime="3.850025098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:00.544656832 +0000 UTC m=+120.898750179" watchObservedRunningTime="2026-09-08 18:36:00.545446771 +0000 UTC m=+120.899540125"
Sep 08 18:36:09.876046 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:09.876005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:36:09.876521 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:09.876131    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 18:36:09.876521 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:09.876188    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:11.876170724 +0000 UTC m=+252.230264052 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : secret "metrics-daemon-secret" not found
Sep 08 18:36:36.490742 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:36.490693    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn" podUID="b0942923-36ad-4be6-afe4-b5c44d783a1c"
Sep 08 18:36:36.499872 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:36.499843    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" podUID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10"
Sep 08 18:36:36.514125 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:36.514100    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nqt6r" podUID="c1654ffb-c4d0-421b-b162-28c6f1377f68"
Sep 08 18:36:36.614776 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:36.614752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:36:36.614878 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:36.614752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:36:36.614942 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:36.614752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:36:38.170201 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:38.170164    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:36:41.398629 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:41.398578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:41.398640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:41.398672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:41.398754    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:41.398767    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64887f6446-mzzw4: secret "image-registry-tls" not found
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:41.398793    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:41.398822    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls podName:180ed34c-d8b9-4d11-a1b0-1d99770a7c10 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:43.398808432 +0000 UTC m=+283.752901744 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls") pod "image-registry-64887f6446-mzzw4" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10") : secret "image-registry-tls" not found
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:41.398752    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:41.398863    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls podName:c1654ffb-c4d0-421b-b162-28c6f1377f68 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:43.398845811 +0000 UTC m=+283.752939143 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls") pod "dns-default-nqt6r" (UID: "c1654ffb-c4d0-421b-b162-28c6f1377f68") : secret "dns-default-metrics-tls" not found
Sep 08 18:36:41.399020 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:36:41.398884    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert podName:b0942923-36ad-4be6-afe4-b5c44d783a1c nodeName:}" failed. No retries permitted until 2026-09-08 18:38:43.3988714 +0000 UTC m=+283.752964744 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6cmn" (UID: "b0942923-36ad-4be6-afe4-b5c44d783a1c") : secret "networking-console-plugin-cert" not found
Sep 08 18:36:53.156655 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:36:53.156619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:38:00.533165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:00.533126    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z" podUID="47fcba70-51a4-4449-a367-35589af8f26c" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Sep 08 18:38:00.818083 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:00.818003    2579 generic.go:356] "Generic (PLEG): container finished" podID="47fcba70-51a4-4449-a367-35589af8f26c" containerID="da90a62858efe042362a3db9a674f6573b635b947c91109a3523acd7b3d06816" exitCode=1
Sep 08 18:38:00.818083 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:00.818051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z" event={"ID":"47fcba70-51a4-4449-a367-35589af8f26c","Type":"ContainerDied","Data":"da90a62858efe042362a3db9a674f6573b635b947c91109a3523acd7b3d06816"}
Sep 08 18:38:00.818427 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:00.818411    2579 scope.go:117] "RemoveContainer" containerID="da90a62858efe042362a3db9a674f6573b635b947c91109a3523acd7b3d06816"
Sep 08 18:38:01.345117 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:01.345082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:01.823291 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:01.823207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z" event={"ID":"47fcba70-51a4-4449-a367-35589af8f26c","Type":"ContainerStarted","Data":"5e5b0b828134913e43e6a6907058f36bed87d38eb918972a87d61e81e1c9fc76"}
Sep 08 18:38:01.823651 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:01.823538    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:38:01.824144 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:01.824129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-685868d9f8-sjk6z"
Sep 08 18:38:11.918180 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:11.918128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:38:11.918674 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:11.918283    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 18:38:11.918674 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:11.918379    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs podName:7a40264b-ad31-4698-926a-5d35143a916f nodeName:}" failed. No retries permitted until 2026-09-08 18:40:13.91835837 +0000 UTC m=+374.272451694 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs") pod "network-metrics-daemon-8mlg8" (UID: "7a40264b-ad31-4698-926a-5d35143a916f") : secret "metrics-daemon-secret" not found
Sep 08 18:38:26.160832 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.160758    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"]
Sep 08 18:38:26.162555 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.162533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.164610 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.164588    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 18:38:26.164729 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.164586    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 18:38:26.164729 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.164590    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:38:26.165169 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.165153    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-q526w\""
Sep 08 18:38:26.165169 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.165158    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 18:38:26.171000 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.170980    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5cb4f5599b-zzlp9"]
Sep 08 18:38:26.172780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.172767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.175320 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.175302    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 18:38:26.175664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.175636    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 18:38:26.176072 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.176048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 18:38:26.176072 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.176048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 18:38:26.176209 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.176057    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7vxmd\""
Sep 08 18:38:26.176465 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.176449    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 18:38:26.176989 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.176970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"]
Sep 08 18:38:26.177793 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.177774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 18:38:26.195378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.195352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cb4f5599b-zzlp9"]
Sep 08 18:38:26.219914 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.219888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-default-certificate\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.220023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.219918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh76f\" (UniqueName: \"kubernetes.io/projected/3700bab8-9882-40a0-a44c-a311bc107d3d-kube-api-access-dh76f\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.220023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.219941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-stats-auth\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.220023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.219981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.220023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.220000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxtbq\" (UniqueName: \"kubernetes.io/projected/07b49476-d0eb-4480-a6bf-314f4417473d-kube-api-access-xxtbq\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.220148 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.220054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3700bab8-9882-40a0-a44c-a311bc107d3d-serving-cert\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.220148 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.220084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3700bab8-9882-40a0-a44c-a311bc107d3d-config\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.220148 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.220119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.265172 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.265141    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"]
Sep 08 18:38:26.266930 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.266912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:26.269542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.269526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 08 18:38:26.269542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.269535    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 08 18:38:26.269691 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.269587    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:38:26.269691 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.269645    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9mkw8\""
Sep 08 18:38:26.278312 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.278292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"]
Sep 08 18:38:26.321412 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3700bab8-9882-40a0-a44c-a311bc107d3d-config\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.321553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:26.321553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-default-certificate\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.321553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxtbq\" (UniqueName: \"kubernetes.io/projected/07b49476-d0eb-4480-a6bf-314f4417473d-kube-api-access-xxtbq\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.321553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3700bab8-9882-40a0-a44c-a311bc107d3d-serving-cert\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.321553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kh22\" (UniqueName: \"kubernetes.io/projected/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-kube-api-access-8kh22\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:26.321553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh76f\" (UniqueName: \"kubernetes.io/projected/3700bab8-9882-40a0-a44c-a311bc107d3d-kube-api-access-dh76f\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.321855 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-stats-auth\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.321855 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.321855 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.321722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.321855 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.321771    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:26.821748661 +0000 UTC m=+267.175841995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : configmap references non-existent config key: service-ca.crt
Sep 08 18:38:26.321855 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.321821    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 18:38:26.322076 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.321882    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:26.821863559 +0000 UTC m=+267.175956885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : secret "router-metrics-certs-default" not found
Sep 08 18:38:26.322076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.322027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3700bab8-9882-40a0-a44c-a311bc107d3d-config\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.323778 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.323750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3700bab8-9882-40a0-a44c-a311bc107d3d-serving-cert\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.324005 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.323986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-default-certificate\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.324050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.323987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-stats-auth\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.328785 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.328765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh76f\" (UniqueName: \"kubernetes.io/projected/3700bab8-9882-40a0-a44c-a311bc107d3d-kube-api-access-dh76f\") pod \"service-ca-operator-859f94c4b-wjdp7\" (UID: \"3700bab8-9882-40a0-a44c-a311bc107d3d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.328901 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.328859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxtbq\" (UniqueName: \"kubernetes.io/projected/07b49476-d0eb-4480-a6bf-314f4417473d-kube-api-access-xxtbq\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.388757 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.388726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:26.422675 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.422599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kh22\" (UniqueName: \"kubernetes.io/projected/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-kube-api-access-8kh22\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:26.422675 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.422668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:26.422865 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.422762    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 18:38:26.422865 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.422808    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls podName:f783aba0-3ae6-4f50-8ad1-66b3041d4dce nodeName:}" failed. No retries permitted until 2026-09-08 18:38:26.922796564 +0000 UTC m=+267.276889875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nvszk" (UID: "f783aba0-3ae6-4f50-8ad1-66b3041d4dce") : secret "samples-operator-tls" not found
Sep 08 18:38:26.432526 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.432502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kh22\" (UniqueName: \"kubernetes.io/projected/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-kube-api-access-8kh22\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:26.471989 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.471952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"
Sep 08 18:38:26.489029 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.489004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:26.736216 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.736141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7"]
Sep 08 18:38:26.741372 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:38:26.741324    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3700bab8_9882_40a0_a44c_a311bc107d3d.slice/crio-920e6c001e4abe68d1b61a99babc6ed73daa32925423f9aec7e7a705eed8845d WatchSource:0}: Error finding container 920e6c001e4abe68d1b61a99babc6ed73daa32925423f9aec7e7a705eed8845d: Status 404 returned error can't find the container with id 920e6c001e4abe68d1b61a99babc6ed73daa32925423f9aec7e7a705eed8845d
Sep 08 18:38:26.825562 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.825532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.825694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.825570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:26.825694 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.825657    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 18:38:26.825694 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.825680    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:27.825660544 +0000 UTC m=+268.179753877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : configmap references non-existent config key: service-ca.crt
Sep 08 18:38:26.825828 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.825704    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:27.825696975 +0000 UTC m=+268.179790285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : secret "router-metrics-certs-default" not found
Sep 08 18:38:26.890670 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.890639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7" event={"ID":"3700bab8-9882-40a0-a44c-a311bc107d3d","Type":"ContainerStarted","Data":"920e6c001e4abe68d1b61a99babc6ed73daa32925423f9aec7e7a705eed8845d"}
Sep 08 18:38:26.926115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:26.926089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:26.926244 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.926226    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 18:38:26.926295 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:26.926287    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls podName:f783aba0-3ae6-4f50-8ad1-66b3041d4dce nodeName:}" failed. No retries permitted until 2026-09-08 18:38:27.926270841 +0000 UTC m=+268.280364170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nvszk" (UID: "f783aba0-3ae6-4f50-8ad1-66b3041d4dce") : secret "samples-operator-tls" not found
Sep 08 18:38:27.834613 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:27.834581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:27.835079 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:27.834715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:27.835079 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:27.834740    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 18:38:27.835079 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:27.834821    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:29.834801053 +0000 UTC m=+270.188894384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : secret "router-metrics-certs-default" not found
Sep 08 18:38:27.835079 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:27.834842    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:29.834833017 +0000 UTC m=+270.188926328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : configmap references non-existent config key: service-ca.crt
Sep 08 18:38:27.935140 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:27.935103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:27.935326 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:27.935221    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 18:38:27.935326 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:27.935278    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls podName:f783aba0-3ae6-4f50-8ad1-66b3041d4dce nodeName:}" failed. No retries permitted until 2026-09-08 18:38:29.93526098 +0000 UTC m=+270.289354292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nvszk" (UID: "f783aba0-3ae6-4f50-8ad1-66b3041d4dce") : secret "samples-operator-tls" not found
Sep 08 18:38:28.558806 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:28.558761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:28.605472 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:28.605429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:28.897137 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:28.897105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7" event={"ID":"3700bab8-9882-40a0-a44c-a311bc107d3d","Type":"ContainerStarted","Data":"32528bad8f431d8b94aad451ee6ee7b422e3fa16b14066d7319cb7e70d671d45"}
Sep 08 18:38:28.910892 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:28.910843    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7" podStartSLOduration=0.871538507 podStartE2EDuration="2.910823661s" totalImagesPullingTime="2.039285154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:28.909546177 +0000 UTC m=+269.263639511" watchObservedRunningTime="2026-09-08 18:38:28.910823661 +0000 UTC m=+269.264916998"
Sep 08 18:38:29.848683 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:29.848645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:29.848683 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:29.848686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:29.848883 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:29.848788    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 18:38:29.848883 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:29.848829    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:33.848809802 +0000 UTC m=+274.202903133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : configmap references non-existent config key: service-ca.crt
Sep 08 18:38:29.848952 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:29.848889    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:33.848874074 +0000 UTC m=+274.202967399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : secret "router-metrics-certs-default" not found
Sep 08 18:38:29.949826 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:29.949793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:29.950277 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:29.949899    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 18:38:29.950277 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:29.949953    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls podName:f783aba0-3ae6-4f50-8ad1-66b3041d4dce nodeName:}" failed. No retries permitted until 2026-09-08 18:38:33.949937278 +0000 UTC m=+274.304030588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nvszk" (UID: "f783aba0-3ae6-4f50-8ad1-66b3041d4dce") : secret "samples-operator-tls" not found
Sep 08 18:38:30.044775 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.044745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:30.097824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.097777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:30.511223 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.511190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:30.568219 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.568189    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:30.663998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.663967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:30.764155 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.764088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:30.864614 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.864583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:30.930408 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.930378    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-92rz7"]
Sep 08 18:38:30.932580 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.932563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:30.934382 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.934336    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 18:38:30.934497 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.934399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 18:38:30.934497 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.934449    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 18:38:30.935033 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.935014    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 18:38:30.935187 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.935068    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q24t4\""
Sep 08 18:38:30.941575 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.941550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-92rz7"]
Sep 08 18:38:30.952135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.952113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:30.957184 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.957166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-crio-socket\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:30.957274 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.957192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:30.957274 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.957228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-data-volume\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:30.957403 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.957335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:30.957463 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:30.957407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kggxf\" (UniqueName: \"kubernetes.io/projected/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-api-access-kggxf\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.006568 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.006540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:31.058832 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.058772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-crio-socket\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.058832 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.058800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.058995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.058839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-data-volume\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.058995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.058893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-crio-socket\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.058995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.058900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.058995 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:31.058905    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:38:31.058995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.058962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kggxf\" (UniqueName: \"kubernetes.io/projected/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-api-access-kggxf\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.058995 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:31.058977    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:31.558958287 +0000 UTC m=+271.913051600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : configmap "kube-rbac-proxy" not found
Sep 08 18:38:31.059293 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:31.058994    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:31.059293 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:31.059045    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:31.559029393 +0000 UTC m=+271.913122721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:31.059293 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.059238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-data-volume\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.066282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.066263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kggxf\" (UniqueName: \"kubernetes.io/projected/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-api-access-kggxf\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.563122 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.563091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.563305 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.563161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:31.563305 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:31.563218    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:38:31.563305 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:31.563292    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:32.563278019 +0000 UTC m=+272.917371343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : configmap "kube-rbac-proxy" not found
Sep 08 18:38:31.563490 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:31.563330    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:31.563490 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:31.563418    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:32.563399808 +0000 UTC m=+272.917493138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:31.697880 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.697849    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:31.798299 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:31.798268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:32.382119 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:32.382083    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:32.434707 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:32.434676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:32.571044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:32.571007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:32.571225 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:32.571124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:32.571225 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:32.571175    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:32.571225 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:32.571196    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:38:32.571412 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:32.571247    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:34.571233109 +0000 UTC m=+274.925326420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : configmap "kube-rbac-proxy" not found
Sep 08 18:38:32.571412 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:32.571262    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:34.571255963 +0000 UTC m=+274.925349274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:32.906278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:32.906248    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rn2xb_eb44a0dc-533c-4750-91aa-766b956e2465/dns-node-resolver/0.log"
Sep 08 18:38:33.213121 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:33.213039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:33.882196 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:33.882159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:33.882196 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:33.882199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:33.882680 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:33.882317    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 18:38:33.882680 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:33.882321    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:41.882304616 +0000 UTC m=+282.236397927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : configmap references non-existent config key: service-ca.crt
Sep 08 18:38:33.882680 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:33.882379    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:41.882364101 +0000 UTC m=+282.236457433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : secret "router-metrics-certs-default" not found
Sep 08 18:38:33.982594 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:33.982550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:33.982767 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:33.982702    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 18:38:33.982767 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:33.982765    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls podName:f783aba0-3ae6-4f50-8ad1-66b3041d4dce nodeName:}" failed. No retries permitted until 2026-09-08 18:38:41.982750122 +0000 UTC m=+282.336843470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nvszk" (UID: "f783aba0-3ae6-4f50-8ad1-66b3041d4dce") : secret "samples-operator-tls" not found
Sep 08 18:38:34.106608 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:34.106581    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kslr8_21b4eada-cc2e-4655-90b8-38e6f9bbcf6a/node-ca/0.log"
Sep 08 18:38:34.586509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:34.586467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:34.586692 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:34.586556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:34.586692 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:34.586587    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:38:34.586692 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:34.586664    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:38.586644865 +0000 UTC m=+278.940738192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : configmap "kube-rbac-proxy" not found
Sep 08 18:38:34.586829 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:34.586696    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:34.586829 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:34.586737    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:38.586724911 +0000 UTC m=+278.940818231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:34.706692 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:34.706664    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-flcd8_e2f54353-feeb-4c43-8b29-9c863b8dd040/serve-healthcheck-canary/0.log"
Sep 08 18:38:35.897260 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:35.897223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:36.107604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:36.107574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:36.208250 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:36.208175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:36.310590 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:36.310556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:36.412564 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:36.412520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:36.595664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:36.595637    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:37.391452 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:37.391423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:38.023581 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:38.023550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:38.191905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:38.191869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:38.398145 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:38.398116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:38.592446 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:38.592405    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:38.619469 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:38.619446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:38.619539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:38.619507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:38.619578 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:38.619560    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:38:38.619624 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:38.619614    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:46.619599138 +0000 UTC m=+286.973692462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : configmap "kube-rbac-proxy" not found
Sep 08 18:38:38.619667 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:38.619631    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:38.619708 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:38.619675    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls podName:1c0dc7ea-4220-4dc8-96e8-b7b534ce098f nodeName:}" failed. No retries permitted until 2026-09-08 18:38:46.61965901 +0000 UTC m=+286.973752341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-92rz7" (UID: "1c0dc7ea-4220-4dc8-96e8-b7b534ce098f") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:38:39.379547 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:39.379519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:39.435216 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:39.435190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:39.615974 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:39.615931    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" podUID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10"
Sep 08 18:38:39.615974 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:39.615966    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn" podUID="b0942923-36ad-4be6-afe4-b5c44d783a1c"
Sep 08 18:38:39.616163 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:39.615967    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nqt6r" podUID="c1654ffb-c4d0-421b-b162-28c6f1377f68"
Sep 08 18:38:39.790440 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:39.790361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:39.847242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:39.847215    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:39.928975 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:39.928952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:38:39.929110 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:39.928984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:38:39.929110 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:39.929093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:38:39.991573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:39.991547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:40.190293 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:40.190261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:40.395433 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:40.395401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:40.590299 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:40.590266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:41.588847 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:41.588807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:41.945389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:41.945288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:41.945389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:41.945325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:41.945778 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:41.945460    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle podName:07b49476-d0eb-4480-a6bf-314f4417473d nodeName:}" failed. No retries permitted until 2026-09-08 18:38:57.945442645 +0000 UTC m=+298.299535962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle") pod "router-default-5cb4f5599b-zzlp9" (UID: "07b49476-d0eb-4480-a6bf-314f4417473d") : configmap references non-existent config key: service-ca.crt
Sep 08 18:38:41.947654 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:41.947634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b49476-d0eb-4480-a6bf-314f4417473d-metrics-certs\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:41.991030 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:41.991004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:42.046563 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.046532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:42.048871 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.048842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f783aba0-3ae6-4f50-8ad1-66b3041d4dce-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nvszk\" (UID: \"f783aba0-3ae6-4f50-8ad1-66b3041d4dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:42.175830 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.175802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"
Sep 08 18:38:42.203859 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.203799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:42.381712 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.381674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:42.441087 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.441059    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk"]
Sep 08 18:38:42.444328 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.444302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:42.589725 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.589689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:42.938562 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:42.938522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk" event={"ID":"f783aba0-3ae6-4f50-8ad1-66b3041d4dce","Type":"ContainerStarted","Data":"4cad7cad0a0909f9d97e785fd308be3ec87656fcf1baf0c74092647cf0576f38"}
Sep 08 18:38:43.459062 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.459027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:38:43.459538 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.459205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:38:43.459538 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.459314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:38:43.461719 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.461698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0942923-36ad-4be6-afe4-b5c44d783a1c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6cmn\" (UID: \"b0942923-36ad-4be6-afe4-b5c44d783a1c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:38:43.461828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.461799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1654ffb-c4d0-421b-b162-28c6f1377f68-metrics-tls\") pod \"dns-default-nqt6r\" (UID: \"c1654ffb-c4d0-421b-b162-28c6f1377f68\") " pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:38:43.461876 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.461835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"image-registry-64887f6446-mzzw4\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") " pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:38:43.532077 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.532046    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xlp48\""
Sep 08 18:38:43.532077 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.532070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xjn8s\""
Sep 08 18:38:43.532278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.532046    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rsh8d\""
Sep 08 18:38:43.540615 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.540589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn"
Sep 08 18:38:43.540727 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.540696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:38:43.540786 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.540739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:38:43.600187 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.600153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:43.754214 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.754145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:43.825148 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.825107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nqt6r"]
Sep 08 18:38:43.828586 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.828570    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x6cmn"]
Sep 08 18:38:43.828755 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.828729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:43.833972 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:43.833952    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64887f6446-mzzw4"]
Sep 08 18:38:44.167409 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:38:44.167384    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1654ffb_c4d0_421b_b162_28c6f1377f68.slice/crio-e9725acb037228e5617dc220634a798228c819c82a87e276125d92a5ed0ab996 WatchSource:0}: Error finding container e9725acb037228e5617dc220634a798228c819c82a87e276125d92a5ed0ab996: Status 404 returned error can't find the container with id e9725acb037228e5617dc220634a798228c819c82a87e276125d92a5ed0ab996
Sep 08 18:38:44.168026 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:38:44.168007    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180ed34c_d8b9_4d11_a1b0_1d99770a7c10.slice/crio-3b325ef5d3d639deeb138c99c7001cb36969bd0eed1bac32ef47f30d4460d194 WatchSource:0}: Error finding container 3b325ef5d3d639deeb138c99c7001cb36969bd0eed1bac32ef47f30d4460d194: Status 404 returned error can't find the container with id 3b325ef5d3d639deeb138c99c7001cb36969bd0eed1bac32ef47f30d4460d194
Sep 08 18:38:44.168989 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:38:44.168955    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0942923_36ad_4be6_afe4_b5c44d783a1c.slice/crio-a09d415c65be26f84b799e60e1c275bf5dfe300770dd08dac34b061a304754aa WatchSource:0}: Error finding container a09d415c65be26f84b799e60e1c275bf5dfe300770dd08dac34b061a304754aa: Status 404 returned error can't find the container with id a09d415c65be26f84b799e60e1c275bf5dfe300770dd08dac34b061a304754aa
Sep 08 18:38:44.192991 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.192916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:44.446315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.446238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:44.946951 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.946916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn" event={"ID":"b0942923-36ad-4be6-afe4-b5c44d783a1c","Type":"ContainerStarted","Data":"a09d415c65be26f84b799e60e1c275bf5dfe300770dd08dac34b061a304754aa"}
Sep 08 18:38:44.949239 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.949207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk" event={"ID":"f783aba0-3ae6-4f50-8ad1-66b3041d4dce","Type":"ContainerStarted","Data":"024cc4122ba6803baf8c84e9ed6b69ac2cc60d75cb2045f86c9d552dcaa2d213"}
Sep 08 18:38:44.949386 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.949266    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk" event={"ID":"f783aba0-3ae6-4f50-8ad1-66b3041d4dce","Type":"ContainerStarted","Data":"7d9a24225ca643ba5c7c93a17139e3877b02533491e7ec0758a5a92d194139a1"}
Sep 08 18:38:44.951248 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.951218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" event={"ID":"180ed34c-d8b9-4d11-a1b0-1d99770a7c10","Type":"ContainerStarted","Data":"abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982"}
Sep 08 18:38:44.951373 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.951278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:38:44.951373 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.951300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" event={"ID":"180ed34c-d8b9-4d11-a1b0-1d99770a7c10","Type":"ContainerStarted","Data":"3b325ef5d3d639deeb138c99c7001cb36969bd0eed1bac32ef47f30d4460d194"}
Sep 08 18:38:44.952762 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.952737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqt6r" event={"ID":"c1654ffb-c4d0-421b-b162-28c6f1377f68","Type":"ContainerStarted","Data":"e9725acb037228e5617dc220634a798228c819c82a87e276125d92a5ed0ab996"}
Sep 08 18:38:44.963332 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.963271    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nvszk" podStartSLOduration=17.247418964 podStartE2EDuration="18.963254491s" totalImagesPullingTime="1.715835527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:44.962439849 +0000 UTC m=+285.316533183" watchObservedRunningTime="2026-09-08 18:38:44.963254491 +0000 UTC m=+285.317347821"
Sep 08 18:38:44.978677 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.978641    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" podStartSLOduration=283.978629602 podStartE2EDuration="4m43.978629602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:44.977803005 +0000 UTC m=+285.331896338" watchObservedRunningTime="2026-09-08 18:38:44.978629602 +0000 UTC m=+285.332722935"
Sep 08 18:38:44.998050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:44.997999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:45.589619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:45.589578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:45.788795 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:45.788707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:45.957493 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:45.957459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn" event={"ID":"b0942923-36ad-4be6-afe4-b5c44d783a1c","Type":"ContainerStarted","Data":"ab8d039c356027cf5f6ef00a410da06c4a6b66413ebe968082f9faded6b4d9b9"}
Sep 08 18:38:45.971149 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:45.971105    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-x6cmn" podStartSLOduration=261.779359408 podStartE2EDuration="4m22.971088729s" totalImagesPullingTime="1.191729321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:45.970387664 +0000 UTC m=+286.324480998" watchObservedRunningTime="2026-09-08 18:38:45.971088729 +0000 UTC m=+286.325182062"
Sep 08 18:38:45.990621 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:45.990589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:46.389808 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.389778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:46.591668 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.591640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:46.688863 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.688772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:46.688863 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.688834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:46.689260 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.689238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:46.691107 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.691084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0dc7ea-4220-4dc8-96e8-b7b534ce098f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-92rz7\" (UID: \"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f\") " pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:46.841486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.841455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-92rz7"
Sep 08 18:38:46.962578 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.962495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqt6r" event={"ID":"c1654ffb-c4d0-421b-b162-28c6f1377f68","Type":"ContainerStarted","Data":"40e0b20e7e9a6c8f0d8ce7e52bb4f505aa301ffeb158384e9f491991d53e33ab"}
Sep 08 18:38:46.962578 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.962551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqt6r" event={"ID":"c1654ffb-c4d0-421b-b162-28c6f1377f68","Type":"ContainerStarted","Data":"42fc48a514eff3fbfac4d315e5746ff6806c5c6d6b2f6933397226b2ab4d73b1"}
Sep 08 18:38:46.979876 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:46.979830    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nqt6r" podStartSLOduration=251.739541815 podStartE2EDuration="4m13.979814094s" totalImagesPullingTime="2.240272279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:46.978662208 +0000 UTC m=+287.332755541" watchObservedRunningTime="2026-09-08 18:38:46.979814094 +0000 UTC m=+287.333907456"
Sep 08 18:38:47.109999 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:47.109967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-92rz7"]
Sep 08 18:38:47.113277 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:38:47.113249    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0dc7ea_4220_4dc8_96e8_b7b534ce098f.slice/crio-0244a22ac8d278ca612972212a01b3a1fe9fcd53ca6eec901a1f0d60ce6e0ed4 WatchSource:0}: Error finding container 0244a22ac8d278ca612972212a01b3a1fe9fcd53ca6eec901a1f0d60ce6e0ed4: Status 404 returned error can't find the container with id 0244a22ac8d278ca612972212a01b3a1fe9fcd53ca6eec901a1f0d60ce6e0ed4
Sep 08 18:38:47.967522 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:47.967494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-92rz7" event={"ID":"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f","Type":"ContainerStarted","Data":"cadbd3f14608d3a0d84a0cf5eb461652e6e22dd6a5dea20fcb65d1fdb628d739"}
Sep 08 18:38:47.967888 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:47.967558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-92rz7" event={"ID":"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f","Type":"ContainerStarted","Data":"706a7d919e4989c5a2c0523d2a05d46d34e19c48bfe9382d46e9d92f1257717e"}
Sep 08 18:38:47.967888 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:47.967579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-92rz7" event={"ID":"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f","Type":"ContainerStarted","Data":"0244a22ac8d278ca612972212a01b3a1fe9fcd53ca6eec901a1f0d60ce6e0ed4"}
Sep 08 18:38:47.967888 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:47.967757    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:38:49.977364 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:49.977260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-92rz7" event={"ID":"1c0dc7ea-4220-4dc8-96e8-b7b534ce098f","Type":"ContainerStarted","Data":"ebdfb6e15aa986f69b4255248f1e6e131cd93976109653b5d0afa25f86b8f219"}
Sep 08 18:38:49.994333 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:49.994287    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-92rz7" podStartSLOduration=17.606049411 podStartE2EDuration="19.994274144s" totalImagesPullingTime="2.388224733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:38:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:49.993567839 +0000 UTC m=+290.347661183" watchObservedRunningTime="2026-09-08 18:38:49.994274144 +0000 UTC m=+290.348367476"
Sep 08 18:38:50.279403 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:50.279321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:50.324103 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:50.324080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:50.386186 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:50.386159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:50.594536 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:50.594508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:50.964793 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:50.964716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:51.035296 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:51.035266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:51.165833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:51.165798    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:51.267440 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:51.267371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:51.389946 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:51.389913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:51.772984 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:51.772959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:51.990989 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:51.990957    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:52.087970 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:52.087939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:52.184334 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:52.184305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:52.281168 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:52.281134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:52.991451 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:52.991420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:53.044772 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:53.044742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:53.548396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:53.548364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:53.648282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:53.648253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:56.157568 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:38:56.157531    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8mlg8" podUID="7a40264b-ad31-4698-926a-5d35143a916f"
Sep 08 18:38:57.974994 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:57.974965    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nqt6r"
Sep 08 18:38:57.984289 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:57.984263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:57.984842 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:57.984823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b49476-d0eb-4480-a6bf-314f4417473d-service-ca-bundle\") pod \"router-default-5cb4f5599b-zzlp9\" (UID: \"07b49476-d0eb-4480-a6bf-314f4417473d\") " pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:58.281799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:58.281707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:58.549930 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:58.549845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cb4f5599b-zzlp9"]
Sep 08 18:38:58.553044 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:38:58.553011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b49476_d0eb_4480_a6bf_314f4417473d.slice/crio-b0110cf9130175a0b6c7e9dbb523a5d9489d195ab920945e95a9c577a5f527da WatchSource:0}: Error finding container b0110cf9130175a0b6c7e9dbb523a5d9489d195ab920945e95a9c577a5f527da: Status 404 returned error can't find the container with id b0110cf9130175a0b6c7e9dbb523a5d9489d195ab920945e95a9c577a5f527da
Sep 08 18:38:58.589903 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:58.589876    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:58.651156 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:58.651011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:38:59.005756 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:59.005718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9" event={"ID":"07b49476-d0eb-4480-a6bf-314f4417473d","Type":"ContainerStarted","Data":"58ad3a1e5e4ef044662d32e2340dad7f3cb1f75e23c2170c31d7c35e705ec103"}
Sep 08 18:38:59.006136 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:59.005776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9" event={"ID":"07b49476-d0eb-4480-a6bf-314f4417473d","Type":"ContainerStarted","Data":"b0110cf9130175a0b6c7e9dbb523a5d9489d195ab920945e95a9c577a5f527da"}
Sep 08 18:38:59.022557 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:59.022514    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9" podStartSLOduration=33.022497769 podStartE2EDuration="33.022497769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:59.021431264 +0000 UTC m=+299.375524598" watchObservedRunningTime="2026-09-08 18:38:59.022497769 +0000 UTC m=+299.376591101"
Sep 08 18:38:59.282856 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:59.282777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:38:59.285133 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:38:59.285112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:39:00.009224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:00.009190    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:39:00.010431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:00.010412    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5cb4f5599b-zzlp9"
Sep 08 18:39:00.042066 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:00.041967    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:39:00.043433 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:00.043325    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:39:00.050367 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:00.050325    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 18:39:00.592077 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:00.592031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:00.692194 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:00.692161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:00.954713 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:00.954613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:01.073713 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.073672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:01.076828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.076805    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"]
Sep 08 18:39:01.079705 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.079689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"
Sep 08 18:39:01.081659 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.081638    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v5dvq\""
Sep 08 18:39:01.081735 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.081668    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 18:39:01.086280 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.086259    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"]
Sep 08 18:39:01.146957 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.146921    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c5ffc697-plqzc"]
Sep 08 18:39:01.149840 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.149822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.153045 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.152058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 18:39:01.153045 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.152704    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 18:39:01.153207 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.153155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vd5bl\""
Sep 08 18:39:01.153258 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.153218    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 18:39:01.153258 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.153246    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 18:39:01.153368 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.153224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 18:39:01.153368 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.153219    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 18:39:01.153475 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.153411    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 18:39:01.158918 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.158896    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c5ffc697-plqzc"]
Sep 08 18:39:01.206214 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.206145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b5dba47-b708-4ec5-8cb3-f006c08f9573-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-qjptq\" (UID: \"8b5dba47-b708-4ec5-8cb3-f006c08f9573\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"
Sep 08 18:39:01.306929 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.306880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-serving-cert\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.307112 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.306956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-oauth-config\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.307112 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.306979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-config\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.307112 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.307021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-oauth-serving-cert\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.307112 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.307077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b5dba47-b708-4ec5-8cb3-f006c08f9573-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-qjptq\" (UID: \"8b5dba47-b708-4ec5-8cb3-f006c08f9573\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"
Sep 08 18:39:01.307313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.307119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2cm\" (UniqueName: \"kubernetes.io/projected/9c7218f5-d21b-4313-9bcf-9367a35bcd46-kube-api-access-sz2cm\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.307313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.307142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-service-ca\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.307313 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:39:01.307181    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 18:39:01.307313 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:39:01.307258    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5dba47-b708-4ec5-8cb3-f006c08f9573-tls-certificates podName:8b5dba47-b708-4ec5-8cb3-f006c08f9573 nodeName:}" failed. No retries permitted until 2026-09-08 18:39:01.807235189 +0000 UTC m=+302.161328519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8b5dba47-b708-4ec5-8cb3-f006c08f9573-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-qjptq" (UID: "8b5dba47-b708-4ec5-8cb3-f006c08f9573") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 18:39:01.407695 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.407663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-oauth-config\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.407695 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.407698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-config\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.407890 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.407830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-oauth-serving-cert\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.407890 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.407885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2cm\" (UniqueName: \"kubernetes.io/projected/9c7218f5-d21b-4313-9bcf-9367a35bcd46-kube-api-access-sz2cm\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.408030 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.407912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-service-ca\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.408030 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.407952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-serving-cert\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.408466 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.408436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-config\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.408606 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.408443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-oauth-serving-cert\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.408606 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.408595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-service-ca\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.410181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.410161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-oauth-config\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.410308 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.410257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-serving-cert\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.414703 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.414684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2cm\" (UniqueName: \"kubernetes.io/projected/9c7218f5-d21b-4313-9bcf-9367a35bcd46-kube-api-access-sz2cm\") pod \"console-78c5ffc697-plqzc\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") " pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.460738 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.460677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:01.735154 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.735077    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c5ffc697-plqzc"]
Sep 08 18:39:01.738123 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:39:01.738096    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7218f5_d21b_4313_9bcf_9367a35bcd46.slice/crio-5071f060b211b8a84c72a7e17ba0e392b5bfaad09068fab7a6205c4c261144f0 WatchSource:0}: Error finding container 5071f060b211b8a84c72a7e17ba0e392b5bfaad09068fab7a6205c4c261144f0: Status 404 returned error can't find the container with id 5071f060b211b8a84c72a7e17ba0e392b5bfaad09068fab7a6205c4c261144f0
Sep 08 18:39:01.740220 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.740200    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:39:01.803715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.803681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:01.811799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.811777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b5dba47-b708-4ec5-8cb3-f006c08f9573-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-qjptq\" (UID: \"8b5dba47-b708-4ec5-8cb3-f006c08f9573\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"
Sep 08 18:39:01.814213 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.814195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b5dba47-b708-4ec5-8cb3-f006c08f9573-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-qjptq\" (UID: \"8b5dba47-b708-4ec5-8cb3-f006c08f9573\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"
Sep 08 18:39:01.989193 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:01.989099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"
Sep 08 18:39:02.017404 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:02.017371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c5ffc697-plqzc" event={"ID":"9c7218f5-d21b-4313-9bcf-9367a35bcd46","Type":"ContainerStarted","Data":"5071f060b211b8a84c72a7e17ba0e392b5bfaad09068fab7a6205c4c261144f0"}
Sep 08 18:39:02.256865 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:02.256786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"]
Sep 08 18:39:02.259927 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:39:02.259899    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5dba47_b708_4ec5_8cb3_f006c08f9573.slice/crio-05ded9033605855ee709549d7b9aff3092d0f436c8ad63739c28a9a9ba62f62e WatchSource:0}: Error finding container 05ded9033605855ee709549d7b9aff3092d0f436c8ad63739c28a9a9ba62f62e: Status 404 returned error can't find the container with id 05ded9033605855ee709549d7b9aff3092d0f436c8ad63739c28a9a9ba62f62e
Sep 08 18:39:03.023509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:03.023474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq" event={"ID":"8b5dba47-b708-4ec5-8cb3-f006c08f9573","Type":"ContainerStarted","Data":"05ded9033605855ee709549d7b9aff3092d0f436c8ad63739c28a9a9ba62f62e"}
Sep 08 18:39:03.547173 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:03.546875    2579 patch_prober.go:28] interesting pod/image-registry-64887f6446-mzzw4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 18:39:03.547173 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:03.546938    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" podUID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 18:39:04.028582 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:04.028526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq" event={"ID":"8b5dba47-b708-4ec5-8cb3-f006c08f9573","Type":"ContainerStarted","Data":"025d52545e889002b85be8ed30368bf9982bd1e7cc04cbc9eaee09da6b6190de"}
Sep 08 18:39:04.028771 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:04.028633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"
Sep 08 18:39:04.034013 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:04.033986    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq"
Sep 08 18:39:04.042664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:04.042623    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qjptq" podStartSLOduration=1.7826786289999998 podStartE2EDuration="3.042610785s" totalImagesPullingTime="1.259932156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:04.041675034 +0000 UTC m=+304.395768368" watchObservedRunningTime="2026-09-08 18:39:04.042610785 +0000 UTC m=+304.396704132"
Sep 08 18:39:05.033856 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.033771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c5ffc697-plqzc" event={"ID":"9c7218f5-d21b-4313-9bcf-9367a35bcd46","Type":"ContainerStarted","Data":"7d79b5758ffdf7dd0efa892287c66a7921aef157aec33316400d5d785e86fc94"}
Sep 08 18:39:05.098963 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.098936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:05.131053 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.130998    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78c5ffc697-plqzc" podStartSLOduration=1.165104335 podStartE2EDuration="4.130982731s" totalImagesPullingTime="2.965878396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:05.04778943 +0000 UTC m=+305.401882765" watchObservedRunningTime="2026-09-08 18:39:05.130982731 +0000 UTC m=+305.485076064"
Sep 08 18:39:05.131606 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.131582    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-j496w"]
Sep 08 18:39:05.133777 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.133762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.135801 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.135776    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 18:39:05.135912 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.135826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 08 18:39:05.135912 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.135840    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 18:39:05.136023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.135932    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zlclj\""
Sep 08 18:39:05.136023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.135936    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 18:39:05.136023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.135978    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 08 18:39:05.143782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.143761    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-j496w"]
Sep 08 18:39:05.240494 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.240463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.240643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.240528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.240643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.240579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.240643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.240605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2jb\" (UniqueName: \"kubernetes.io/projected/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-kube-api-access-kl2jb\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.341614 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.341585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.341741 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.341651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.341741 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.341676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.341860 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.341838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2jb\" (UniqueName: \"kubernetes.io/projected/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-kube-api-access-kl2jb\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.342325 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.342305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.343878 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.343856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.344330 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.344313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.348860 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.348836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2jb\" (UniqueName: \"kubernetes.io/projected/6d2bb26d-6a47-42d9-84be-14b2bfe31a54-kube-api-access-kl2jb\") pod \"prometheus-operator-6f4946db56-j496w\" (UID: \"6d2bb26d-6a47-42d9-84be-14b2bfe31a54\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.442848 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.442817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w"
Sep 08 18:39:05.708581 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.708382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-j496w"]
Sep 08 18:39:05.710706 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:39:05.710679    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2bb26d_6a47_42d9_84be_14b2bfe31a54.slice/crio-d9971c3a6a8bc154c5f81501f4397469b63013b3b26c36e89cbce0aa1dda828e WatchSource:0}: Error finding container d9971c3a6a8bc154c5f81501f4397469b63013b3b26c36e89cbce0aa1dda828e: Status 404 returned error can't find the container with id d9971c3a6a8bc154c5f81501f4397469b63013b3b26c36e89cbce0aa1dda828e
Sep 08 18:39:05.962725 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:05.962658    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:39:06.037843 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:06.037811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w" event={"ID":"6d2bb26d-6a47-42d9-84be-14b2bfe31a54","Type":"ContainerStarted","Data":"d9971c3a6a8bc154c5f81501f4397469b63013b3b26c36e89cbce0aa1dda828e"}
Sep 08 18:39:06.548978 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:06.548940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:07.087598 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:07.087563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:07.189663 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:07.189600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:07.289597 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:07.289509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:07.585463 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:07.585435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:07.685633 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:07.685603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:08.047524 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:08.047439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w" event={"ID":"6d2bb26d-6a47-42d9-84be-14b2bfe31a54","Type":"ContainerStarted","Data":"369ae90b79647db9781f9fac2c387ec36858beef31a71673a2d65a66bf17f9ed"}
Sep 08 18:39:08.047665 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:08.047537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w" event={"ID":"6d2bb26d-6a47-42d9-84be-14b2bfe31a54","Type":"ContainerStarted","Data":"8a6fa2aa726b0db17263b97042abd1da492760276de9fc9eec5a70d78a89de48"}
Sep 08 18:39:08.061445 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:08.061398    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-j496w" podStartSLOduration=1.754485049 podStartE2EDuration="3.061382307s" totalImagesPullingTime="1.306897258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:08.060024291 +0000 UTC m=+308.414117623" watchObservedRunningTime="2026-09-08 18:39:08.061382307 +0000 UTC m=+308.415475640"
Sep 08 18:39:08.157272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:08.157242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:39:08.541317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:08.541283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:08.589567 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:08.589535    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:09.230813 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.230784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:09.327628 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.327597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:09.456867 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.456835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:09.476603 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.476580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jx7ch"]
Sep 08 18:39:09.479911 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.479895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.482050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.481995    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 18:39:09.482140 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.482071    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 18:39:09.482537 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.482516    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rfb7z\""
Sep 08 18:39:09.482673 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.482602    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 18:39:09.551730 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.551699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:09.578837 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.578810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3740c39d-855a-4039-98b5-c410709e9e9e-metrics-client-ca\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.578949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.578851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.578949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.578873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-root\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.578949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.578936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.579076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.578973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-wtmp\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.579076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.578995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbtd\" (UniqueName: \"kubernetes.io/projected/3740c39d-855a-4039-98b5-c410709e9e9e-kube-api-access-4tbtd\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.579076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.579011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-textfile\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.579076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.579068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-tls\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.579213 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.579112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-sys\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680296 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-sys\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680441 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3740c39d-855a-4039-98b5-c410709e9e9e-metrics-client-ca\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680441 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680441 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-root\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680441 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680441 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-sys\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-wtmp\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbtd\" (UniqueName: \"kubernetes.io/projected/3740c39d-855a-4039-98b5-c410709e9e9e-kube-api-access-4tbtd\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-root\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-wtmp\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-textfile\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-tls\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.680938 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:39:09.680718    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 08 18:39:09.680938 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:39:09.680784    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-tls podName:3740c39d-855a-4039-98b5-c410709e9e9e nodeName:}" failed. No retries permitted until 2026-09-08 18:39:10.180763186 +0000 UTC m=+310.534856515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-tls") pod "node-exporter-jx7ch" (UID: "3740c39d-855a-4039-98b5-c410709e9e9e") : secret "node-exporter-tls" not found
Sep 08 18:39:09.680938 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-textfile\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.681062 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3740c39d-855a-4039-98b5-c410709e9e9e-metrics-client-ca\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.681062 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.680978    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.682685 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.682663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.690055 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.690029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbtd\" (UniqueName: \"kubernetes.io/projected/3740c39d-855a-4039-98b5-c410709e9e9e-kube-api-access-4tbtd\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:09.748958 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.748885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:09.892714 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:09.892667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:10.010895 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.010814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:10.108358 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.108316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:10.183854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.183825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-tls\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:10.186010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.185989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3740c39d-855a-4039-98b5-c410709e9e9e-node-exporter-tls\") pod \"node-exporter-jx7ch\" (UID: \"3740c39d-855a-4039-98b5-c410709e9e9e\") " pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:10.389272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.389242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jx7ch"
Sep 08 18:39:10.401428 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:39:10.401392    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3740c39d_855a_4039_98b5_c410709e9e9e.slice/crio-43c3be7c938e775d9d1d1016ad63f2f62d05a739a4cb4f1b1cd8a08e31caa0ce WatchSource:0}: Error finding container 43c3be7c938e775d9d1d1016ad63f2f62d05a739a4cb4f1b1cd8a08e31caa0ce: Status 404 returned error can't find the container with id 43c3be7c938e775d9d1d1016ad63f2f62d05a739a4cb4f1b1cd8a08e31caa0ce
Sep 08 18:39:10.445697 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.445665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:10.515173 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.515144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:10.523144 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.523120    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:39:10.527729 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.527706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.529685 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.529665    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 18:39:10.530044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.530027    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 18:39:10.530379 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.530364    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gkjbf\""
Sep 08 18:39:10.530914 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.530879    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 18:39:10.531048 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.531028    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 18:39:10.531206 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.531188    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 18:39:10.531265 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.531237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 18:39:10.531326 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.531188    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 18:39:10.531632 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.531465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 18:39:10.531803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.531774    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 18:39:10.542096 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.542075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:39:10.587374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-volume\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587611 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-tls-assets\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587611 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587611 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587611 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587820 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gcwz\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-kube-api-access-8gcwz\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587820 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-out\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587820 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587820 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.587820 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.588031 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-web-config\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.588031 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.587859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.661391 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.661286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:10.688538 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.688700 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gcwz\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-kube-api-access-8gcwz\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.688700 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-out\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.688700 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.688700 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.688913 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.688966 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-web-config\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.689016 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.689016 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.688996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.689123 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.689033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-volume\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.689123 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.689079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-tls-assets\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.689123 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.689120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.689273 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.689144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.689534 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.689506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.689666 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:39:10.689627    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle podName:322e1065-b5a3-4c06-ab36-1c01d2a53a01 nodeName:}" failed. No retries permitted until 2026-09-08 18:39:11.18960408 +0000 UTC m=+311.543697406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01") : configmap references non-existent config key: ca-bundle.crt
Sep 08 18:39:10.690140 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.690119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.690377 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:39:10.690202    2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 08 18:39:10.690477 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:39:10.690456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls podName:322e1065-b5a3-4c06-ab36-1c01d2a53a01 nodeName:}" failed. No retries permitted until 2026-09-08 18:39:11.190436544 +0000 UTC m=+311.544529863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01") : secret "alertmanager-main-tls" not found
Sep 08 18:39:10.691890 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.691860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.691994 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.691964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.692074 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.692038    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.692132 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.692105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-volume\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.692173 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.692110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-out\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.692409 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.692389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.693472 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.693453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-tls-assets\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.693558 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.693539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-web-config\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.698523 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.698503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gcwz\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-kube-api-access-8gcwz\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:10.817596 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:10.817560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:11.058889 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.058811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jx7ch" event={"ID":"3740c39d-855a-4039-98b5-c410709e9e9e","Type":"ContainerStarted","Data":"43c3be7c938e775d9d1d1016ad63f2f62d05a739a4cb4f1b1cd8a08e31caa0ce"}
Sep 08 18:39:11.194094 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.194056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:11.194261 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.194107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:11.195081 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.195053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:11.196846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.196824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:11.438128 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.438097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:39:11.461856 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.461825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:11.462037 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.461896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:11.467786 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.467766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:11.722633 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:11.722605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:39:11.725997 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:39:11.725973    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322e1065_b5a3_4c06_ab36_1c01d2a53a01.slice/crio-7cb98176c0cdeab115499c4a3a7d1aa290063fab28206086f479fdf26bb67994 WatchSource:0}: Error finding container 7cb98176c0cdeab115499c4a3a7d1aa290063fab28206086f479fdf26bb67994: Status 404 returned error can't find the container with id 7cb98176c0cdeab115499c4a3a7d1aa290063fab28206086f479fdf26bb67994
Sep 08 18:39:12.063378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:12.063278    2579 generic.go:356] "Generic (PLEG): container finished" podID="3740c39d-855a-4039-98b5-c410709e9e9e" containerID="ff7b0dfe626274cfd30658ce182615c1470528f0d91033523bbef819a33ef3a1" exitCode=0
Sep 08 18:39:12.063535 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:12.063368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jx7ch" event={"ID":"3740c39d-855a-4039-98b5-c410709e9e9e","Type":"ContainerDied","Data":"ff7b0dfe626274cfd30658ce182615c1470528f0d91033523bbef819a33ef3a1"}
Sep 08 18:39:12.064530 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:12.064510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerStarted","Data":"7cb98176c0cdeab115499c4a3a7d1aa290063fab28206086f479fdf26bb67994"}
Sep 08 18:39:12.070540 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:12.070520    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:12.463992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:12.463958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:12.564536 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:12.564500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:12.665733 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:12.665695    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:13.038387 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:13.038353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:13.070314 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:13.070275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jx7ch" event={"ID":"3740c39d-855a-4039-98b5-c410709e9e9e","Type":"ContainerStarted","Data":"bf463445ca7798d709f3a5d6538fdd991f2e347ee75e8e902fb2785efa10b3ac"}
Sep 08 18:39:13.070473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:13.070336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jx7ch" event={"ID":"3740c39d-855a-4039-98b5-c410709e9e9e","Type":"ContainerStarted","Data":"e1c6762d6e8deb6bca6201fa63280122d20607d81243f4f0699836a884cfaaca"}
Sep 08 18:39:13.071798 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:13.071775    2579 generic.go:356] "Generic (PLEG): container finished" podID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerID="5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322" exitCode=0
Sep 08 18:39:13.071897 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:13.071866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerDied","Data":"5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322"}
Sep 08 18:39:13.088365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:13.088273    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jx7ch" podStartSLOduration=3.022170426 podStartE2EDuration="4.088260221s" totalImagesPullingTime="996.091541ms" totalInitContainerRuntime="69.998254ms" isStatefulPod=true podCreationTimestamp="2026-09-08 18:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:13.08791374 +0000 UTC m=+313.442007073" watchObservedRunningTime="2026-09-08 18:39:13.088260221 +0000 UTC m=+313.442353554"
Sep 08 18:39:13.556092 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:13.556014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:14.259317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:14.259281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:14.354610 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:14.354576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:14.620172 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:14.620144    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64887f6446-mzzw4"]
Sep 08 18:39:14.688596 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:14.688562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:14.795459 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:14.795424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:14.893435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:14.893401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:15.086546 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.086517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerStarted","Data":"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d"}
Sep 08 18:39:15.086693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.086581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerStarted","Data":"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df"}
Sep 08 18:39:15.086693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.086610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerStarted","Data":"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366"}
Sep 08 18:39:15.086693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.086634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerStarted","Data":"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229"}
Sep 08 18:39:15.086693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.086655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerStarted","Data":"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb"}
Sep 08 18:39:15.727947 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.727914    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:39:15.733263 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.733237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.736135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.735780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 18:39:15.736135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.735985    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fl8m9nac3ktn9\""
Sep 08 18:39:15.736135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.736031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c87rq\""
Sep 08 18:39:15.736382 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.736237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 18:39:15.736382 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.736273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 18:39:15.737010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.736716    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 18:39:15.737010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.736760    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 18:39:15.737010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.736790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 18:39:15.737010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.736960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 18:39:15.737255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.737073    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 18:39:15.737255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.737093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:15.737255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.737112    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 18:39:15.737255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.737138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 18:39:15.737255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.737216    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 18:39:15.737255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.737223    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 18:39:15.737255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.737231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 18:39:15.737722 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.737457    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 18:39:15.740954 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.740930    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 18:39:15.762481 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.762450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:39:15.837297 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837297 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962gq\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-kube-api-access-962gq\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837587 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837587 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837587 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837587 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837587 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.837828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.838050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.838050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.838050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.838050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.838050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.837983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.838050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.838031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.838286 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.838056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.938869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.938841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.938974 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.938880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.938974 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.938908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939064 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939368 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939368 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939525 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939525 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939525 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962gq\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-kube-api-access-962gq\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.939682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940042 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940042 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940042 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940042 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940042 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940042 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.939816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940360 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.940104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940360 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.940245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940530 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.940508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.940905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.940875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.943207 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.943176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.943537 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.943490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.943632 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.943584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.944175 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.944019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.944175 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.944132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.944417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.944379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.945817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.945779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.946772 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.946649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.947174 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.947127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.948087 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.948057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.948501 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.948481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.948728 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.948706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-962gq\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-kube-api-access-962gq\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.949357 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.949327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config\") pod \"prometheus-k8s-0\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:15.953453 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:15.953423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:16.004743 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.004701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:16.045856 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.045828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:16.097270 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.097181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerStarted","Data":"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330"}
Sep 08 18:39:16.123961 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.123888    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.4665158480000002 podStartE2EDuration="6.123863293s" totalImagesPullingTime="3.64164855s" totalInitContainerRuntime="15.698895ms" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:16.121291457 +0000 UTC m=+316.475384791" watchObservedRunningTime="2026-09-08 18:39:16.123863293 +0000 UTC m=+316.477956625"
Sep 08 18:39:16.333671 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.333641    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:39:16.336931 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:39:16.336906    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c8b639_1054_43ca_a70d_bbf094b8e5dd.slice/crio-ee150a989223f950db4e51db4011533399843664c28b00542f25d6dc0f5a4c0b WatchSource:0}: Error finding container ee150a989223f950db4e51db4011533399843664c28b00542f25d6dc0f5a4c0b: Status 404 returned error can't find the container with id ee150a989223f950db4e51db4011533399843664c28b00542f25d6dc0f5a4c0b
Sep 08 18:39:16.404433 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.404404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:16.477146 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.477118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:16.649917 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.649835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:16.754014 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:16.753976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:17.102918 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:17.102882    2579 generic.go:356] "Generic (PLEG): container finished" podID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766" exitCode=0
Sep 08 18:39:17.103085 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:17.102983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerDied","Data":"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"}
Sep 08 18:39:17.103085 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:17.103050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerStarted","Data":"ee150a989223f950db4e51db4011533399843664c28b00542f25d6dc0f5a4c0b"}
Sep 08 18:39:21.123876 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:21.123838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerStarted","Data":"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"}
Sep 08 18:39:21.124283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:21.123924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerStarted","Data":"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"}
Sep 08 18:39:21.998664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:21.998639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:22.098675 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:22.098646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:22.132757 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:22.132729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerStarted","Data":"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"}
Sep 08 18:39:22.133055 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:22.132793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerStarted","Data":"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"}
Sep 08 18:39:23.139413 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:23.139378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerStarted","Data":"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"}
Sep 08 18:39:23.139804 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:23.139454    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerStarted","Data":"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"}
Sep 08 18:39:23.164905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:23.164855    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.394331083 podStartE2EDuration="8.164836619s" totalImagesPullingTime="4.751194033s" totalInitContainerRuntime="19.311503ms" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:23.16251237 +0000 UTC m=+323.516605702" watchObservedRunningTime="2026-09-08 18:39:23.164836619 +0000 UTC m=+323.518929944"
Sep 08 18:39:23.550218 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:23.550136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:23.604461 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:23.604427    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:26.046354 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:26.046301    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:39:27.083916 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:27.083887    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c5ffc697-plqzc"]
Sep 08 18:39:28.794853 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:28.794808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:28.897484 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:28.897456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:29.194727 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:29.194697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:29.295031 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:29.295004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:30.100776 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:30.100746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:30.152734 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:30.152705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:31.767160 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:31.767127    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:38.556152 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:38.556093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:38.608427 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:38.608399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:39.646376 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:39.646321    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" podUID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10" containerName="registry" containerID="cri-o://abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982" gracePeriod=30
Sep 08 18:39:39.886848 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:39.886824    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:39:40.054473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") pod \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") "
Sep 08 18:39:40.054473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-trusted-ca\") pod \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") "
Sep 08 18:39:40.054473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-installation-pull-secrets\") pod \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") "
Sep 08 18:39:40.054473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-ca-trust-extracted\") pod \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") "
Sep 08 18:39:40.054728 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054668    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-image-registry-private-configuration\") pod \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") "
Sep 08 18:39:40.054786 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054725    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fswzv\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-kube-api-access-fswzv\") pod \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") "
Sep 08 18:39:40.054839 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054792    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-certificates\") pod \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") "
Sep 08 18:39:40.054839 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054828    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-bound-sa-token\") pod \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\" (UID: \"180ed34c-d8b9-4d11-a1b0-1d99770a7c10\") "
Sep 08 18:39:40.054938 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.054861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "180ed34c-d8b9-4d11-a1b0-1d99770a7c10" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:40.055182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.055144    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-trusted-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:40.055306 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.055244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "180ed34c-d8b9-4d11-a1b0-1d99770a7c10" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:40.057119 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.057079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "180ed34c-d8b9-4d11-a1b0-1d99770a7c10" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:40.057223 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.057178    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "180ed34c-d8b9-4d11-a1b0-1d99770a7c10" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:40.057223 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.057207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-kube-api-access-fswzv" (OuterVolumeSpecName: "kube-api-access-fswzv") pod "180ed34c-d8b9-4d11-a1b0-1d99770a7c10" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10"). InnerVolumeSpecName "kube-api-access-fswzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:40.057327 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.057222    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "180ed34c-d8b9-4d11-a1b0-1d99770a7c10" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:40.057327 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.057277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "180ed34c-d8b9-4d11-a1b0-1d99770a7c10" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:40.063430 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.063397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "180ed34c-d8b9-4d11-a1b0-1d99770a7c10" (UID: "180ed34c-d8b9-4d11-a1b0-1d99770a7c10"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:40.155773 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.155743    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-ca-trust-extracted\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:40.155773 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.155768    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-image-registry-private-configuration\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:40.155957 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.155781    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fswzv\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-kube-api-access-fswzv\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:40.155957 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.155794    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-certificates\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:40.155957 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.155805    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-bound-sa-token\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:40.155957 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.155819    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-registry-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:40.155957 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.155828    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/180ed34c-d8b9-4d11-a1b0-1d99770a7c10-installation-pull-secrets\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:40.203330 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.203298    2579 generic.go:356] "Generic (PLEG): container finished" podID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10" containerID="abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982" exitCode=0
Sep 08 18:39:40.203453 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.203364    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64887f6446-mzzw4"
Sep 08 18:39:40.203453 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.203371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" event={"ID":"180ed34c-d8b9-4d11-a1b0-1d99770a7c10","Type":"ContainerDied","Data":"abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982"}
Sep 08 18:39:40.203453 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.203416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64887f6446-mzzw4" event={"ID":"180ed34c-d8b9-4d11-a1b0-1d99770a7c10","Type":"ContainerDied","Data":"3b325ef5d3d639deeb138c99c7001cb36969bd0eed1bac32ef47f30d4460d194"}
Sep 08 18:39:40.203453 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.203442    2579 scope.go:117] "RemoveContainer" containerID="abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982"
Sep 08 18:39:40.211366 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.211330    2579 scope.go:117] "RemoveContainer" containerID="abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982"
Sep 08 18:39:40.211650 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:39:40.211627    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982\": container with ID starting with abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982 not found: ID does not exist" containerID="abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982"
Sep 08 18:39:40.211710 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.211660    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982"} err="failed to get container status \"abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982\": rpc error: code = NotFound desc = could not find container \"abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982\": container with ID starting with abd6e4d24251fa4bf82c710a60c7a98848d3cc639c3d9ca924ea4c6c58e5a982 not found: ID does not exist"
Sep 08 18:39:40.216638 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.216616    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64887f6446-mzzw4"]
Sep 08 18:39:40.221626 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:40.221606    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64887f6446-mzzw4"]
Sep 08 18:39:42.161957 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:42.161918    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10" path="/var/lib/kubelet/pods/180ed34c-d8b9-4d11-a1b0-1d99770a7c10/volumes"
Sep 08 18:39:52.103383 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.103303    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78c5ffc697-plqzc" podUID="9c7218f5-d21b-4313-9bcf-9367a35bcd46" containerName="console" containerID="cri-o://7d79b5758ffdf7dd0efa892287c66a7921aef157aec33316400d5d785e86fc94" gracePeriod=15
Sep 08 18:39:52.250650 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.250619    2579 generic.go:356] "Generic (PLEG): container finished" podID="9c7218f5-d21b-4313-9bcf-9367a35bcd46" containerID="7d79b5758ffdf7dd0efa892287c66a7921aef157aec33316400d5d785e86fc94" exitCode=0
Sep 08 18:39:52.250799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.250673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c5ffc697-plqzc" event={"ID":"9c7218f5-d21b-4313-9bcf-9367a35bcd46","Type":"ContainerDied","Data":"7d79b5758ffdf7dd0efa892287c66a7921aef157aec33316400d5d785e86fc94"}
Sep 08 18:39:52.359711 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.359649    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:52.467091 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.467054    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-oauth-config\") pod \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") "
Sep 08 18:39:52.467091 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.467094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-serving-cert\") pod \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") "
Sep 08 18:39:52.467365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.467121    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz2cm\" (UniqueName: \"kubernetes.io/projected/9c7218f5-d21b-4313-9bcf-9367a35bcd46-kube-api-access-sz2cm\") pod \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") "
Sep 08 18:39:52.467365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.467158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-oauth-serving-cert\") pod \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") "
Sep 08 18:39:52.467365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.467185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-service-ca\") pod \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") "
Sep 08 18:39:52.467365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.467203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-config\") pod \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\" (UID: \"9c7218f5-d21b-4313-9bcf-9367a35bcd46\") "
Sep 08 18:39:52.467674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.467643    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-service-ca" (OuterVolumeSpecName: "service-ca") pod "9c7218f5-d21b-4313-9bcf-9367a35bcd46" (UID: "9c7218f5-d21b-4313-9bcf-9367a35bcd46"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:52.467735 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.467651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-config" (OuterVolumeSpecName: "console-config") pod "9c7218f5-d21b-4313-9bcf-9367a35bcd46" (UID: "9c7218f5-d21b-4313-9bcf-9367a35bcd46"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:52.469418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.469388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9c7218f5-d21b-4313-9bcf-9367a35bcd46" (UID: "9c7218f5-d21b-4313-9bcf-9367a35bcd46"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:52.469521 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.469498    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7218f5-d21b-4313-9bcf-9367a35bcd46-kube-api-access-sz2cm" (OuterVolumeSpecName: "kube-api-access-sz2cm") pod "9c7218f5-d21b-4313-9bcf-9367a35bcd46" (UID: "9c7218f5-d21b-4313-9bcf-9367a35bcd46"). InnerVolumeSpecName "kube-api-access-sz2cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:52.471566 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.471537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9c7218f5-d21b-4313-9bcf-9367a35bcd46" (UID: "9c7218f5-d21b-4313-9bcf-9367a35bcd46"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:52.471927 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.471906    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9c7218f5-d21b-4313-9bcf-9367a35bcd46" (UID: "9c7218f5-d21b-4313-9bcf-9367a35bcd46"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:52.568542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.568505    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-serving-cert\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:52.568542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.568532    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz2cm\" (UniqueName: \"kubernetes.io/projected/9c7218f5-d21b-4313-9bcf-9367a35bcd46-kube-api-access-sz2cm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:52.568542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.568542    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-oauth-serving-cert\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:52.568542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.568553    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:52.568806 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.568562    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-config\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:52.568806 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:52.568576    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c7218f5-d21b-4313-9bcf-9367a35bcd46-console-oauth-config\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:39:53.255888 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:53.255862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c5ffc697-plqzc"
Sep 08 18:39:53.256251 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:53.255855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c5ffc697-plqzc" event={"ID":"9c7218f5-d21b-4313-9bcf-9367a35bcd46","Type":"ContainerDied","Data":"5071f060b211b8a84c72a7e17ba0e392b5bfaad09068fab7a6205c4c261144f0"}
Sep 08 18:39:53.256251 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:53.255994    2579 scope.go:117] "RemoveContainer" containerID="7d79b5758ffdf7dd0efa892287c66a7921aef157aec33316400d5d785e86fc94"
Sep 08 18:39:53.274943 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:53.274919    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c5ffc697-plqzc"]
Sep 08 18:39:53.282424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:53.280963    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78c5ffc697-plqzc"]
Sep 08 18:39:53.546811 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:53.546739    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:53.599055 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:53.599017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:39:54.163391 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:39:54.163333    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c7218f5-d21b-4313-9bcf-9367a35bcd46" path="/var/lib/kubelet/pods/9c7218f5-d21b-4313-9bcf-9367a35bcd46/volumes"
Sep 08 18:40:00.283203 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:00.283171    2579 generic.go:356] "Generic (PLEG): container finished" podID="3700bab8-9882-40a0-a44c-a311bc107d3d" containerID="32528bad8f431d8b94aad451ee6ee7b422e3fa16b14066d7319cb7e70d671d45" exitCode=0
Sep 08 18:40:00.283609 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:00.283241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7" event={"ID":"3700bab8-9882-40a0-a44c-a311bc107d3d","Type":"ContainerDied","Data":"32528bad8f431d8b94aad451ee6ee7b422e3fa16b14066d7319cb7e70d671d45"}
Sep 08 18:40:00.283609 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:00.283552    2579 scope.go:117] "RemoveContainer" containerID="32528bad8f431d8b94aad451ee6ee7b422e3fa16b14066d7319cb7e70d671d45"
Sep 08 18:40:01.289284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:01.289249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-wjdp7" event={"ID":"3700bab8-9882-40a0-a44c-a311bc107d3d","Type":"ContainerStarted","Data":"58b27cb23217181990116e65829334245ba22ef052f29226afcaad73276f0d0d"}
Sep 08 18:40:13.945403 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:13.945336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:40:13.947644 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:13.947612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a40264b-ad31-4698-926a-5d35143a916f-metrics-certs\") pod \"network-metrics-daemon-8mlg8\" (UID: \"7a40264b-ad31-4698-926a-5d35143a916f\") " pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:40:14.159864 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:14.159838    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z79bg\""
Sep 08 18:40:14.168262 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:14.168241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mlg8"
Sep 08 18:40:14.434586 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:14.434551    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8mlg8"]
Sep 08 18:40:14.437828 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:40:14.437803    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a40264b_ad31_4698_926a_5d35143a916f.slice/crio-2a795002318558394007cf57659bfe5653da85dca5e712a8fda5076fa0e8b537 WatchSource:0}: Error finding container 2a795002318558394007cf57659bfe5653da85dca5e712a8fda5076fa0e8b537: Status 404 returned error can't find the container with id 2a795002318558394007cf57659bfe5653da85dca5e712a8fda5076fa0e8b537
Sep 08 18:40:15.348510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:15.348469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mlg8" event={"ID":"7a40264b-ad31-4698-926a-5d35143a916f","Type":"ContainerStarted","Data":"2a795002318558394007cf57659bfe5653da85dca5e712a8fda5076fa0e8b537"}
Sep 08 18:40:16.046746 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:16.046658    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:16.061956 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:16.061922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:16.353497 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:16.353461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mlg8" event={"ID":"7a40264b-ad31-4698-926a-5d35143a916f","Type":"ContainerStarted","Data":"dcb5ba98c9394739bcfe619053fb99964899b85fc547f1a6d1be0a28314e9d9e"}
Sep 08 18:40:16.353879 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:16.353527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mlg8" event={"ID":"7a40264b-ad31-4698-926a-5d35143a916f","Type":"ContainerStarted","Data":"acfbc25a1762a324398e4b147113eb6ab83eb3a9507a436f63efea1d3b5b6174"}
Sep 08 18:40:16.367855 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:16.367792    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8mlg8" podStartSLOduration=375.332727689 podStartE2EDuration="6m16.367771077s" totalImagesPullingTime="1.035043388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:16.365881207 +0000 UTC m=+376.719974562" watchObservedRunningTime="2026-09-08 18:40:16.367771077 +0000 UTC m=+376.721864410"
Sep 08 18:40:16.369317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:16.369289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:16.501664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:16.501628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:16.549170 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:16.549137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:18.309600 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:18.309562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:18.393728 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:18.393693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:28.729431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:28.729380    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:29.111766 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:29.111728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:29.775096 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:29.775063    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:40:29.775531 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:29.775367    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="alertmanager" containerID="cri-o://113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb" gracePeriod=120
Sep 08 18:40:29.775531 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:29.775447    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy-metric" containerID="cri-o://a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d" gracePeriod=120
Sep 08 18:40:29.775531 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:29.775490    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="prom-label-proxy" containerID="cri-o://ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330" gracePeriod=120
Sep 08 18:40:29.775531 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:29.775474    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="config-reloader" containerID="cri-o://dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229" gracePeriod=120
Sep 08 18:40:29.775755 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:29.775461    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy" containerID="cri-o://180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df" gracePeriod=120
Sep 08 18:40:29.775755 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:29.775461    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy-web" containerID="cri-o://6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366" gracePeriod=120
Sep 08 18:40:30.411139 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:30.411096    2579 generic.go:356] "Generic (PLEG): container finished" podID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerID="ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330" exitCode=0
Sep 08 18:40:30.411139 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:30.411137    2579 generic.go:356] "Generic (PLEG): container finished" podID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerID="180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df" exitCode=0
Sep 08 18:40:30.411378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:30.411148    2579 generic.go:356] "Generic (PLEG): container finished" podID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerID="dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229" exitCode=0
Sep 08 18:40:30.411378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:30.411157    2579 generic.go:356] "Generic (PLEG): container finished" podID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerID="113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb" exitCode=0
Sep 08 18:40:30.411378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:30.411146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerDied","Data":"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330"}
Sep 08 18:40:30.411378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:30.411250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerDied","Data":"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df"}
Sep 08 18:40:30.411378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:30.411285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerDied","Data":"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229"}
Sep 08 18:40:30.411378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:30.411312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerDied","Data":"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb"}
Sep 08 18:40:31.020165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.020144    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.193010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.192923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-volume\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.192961    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-web\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.192978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-cluster-tls-config\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gcwz\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-kube-api-access-8gcwz\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-tls-assets\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193124    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193152    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-out\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-metrics-client-ca\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193234    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-main-db\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193295    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-metric\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193414    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-web-config\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193440    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls\") pod \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\" (UID: \"322e1065-b5a3-4c06-ab36-1c01d2a53a01\") "
Sep 08 18:40:31.193833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193625    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:31.193833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.193710    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.195495 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.195456    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:31.196125 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.195816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:31.196125 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.196014    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-volume" (OuterVolumeSpecName: "config-volume") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:31.196125 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.196022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:40:31.196125 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.196031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:31.196125 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.196084    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:40:31.196467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.196318    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-kube-api-access-8gcwz" (OuterVolumeSpecName: "kube-api-access-8gcwz") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "kube-api-access-8gcwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:40:31.196620 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.196584    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:31.197284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.197257    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-out" (OuterVolumeSpecName: "config-out") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:40:31.198053 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.198031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:31.200149 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.200053    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:31.207310 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.207290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-web-config" (OuterVolumeSpecName: "web-config") pod "322e1065-b5a3-4c06-ab36-1c01d2a53a01" (UID: "322e1065-b5a3-4c06-ab36-1c01d2a53a01"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:31.294085 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294054    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-alertmanager-main-db\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294085 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294082    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294085 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294092    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294122    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-web-config\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294133    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-main-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294143    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294152    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294161    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/322e1065-b5a3-4c06-ab36-1c01d2a53a01-cluster-tls-config\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294169    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gcwz\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-kube-api-access-8gcwz\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294177    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/322e1065-b5a3-4c06-ab36-1c01d2a53a01-tls-assets\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294184    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/322e1065-b5a3-4c06-ab36-1c01d2a53a01-config-out\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.294278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.294192    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322e1065-b5a3-4c06-ab36-1c01d2a53a01-metrics-client-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:31.417786 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.417743    2579 generic.go:356] "Generic (PLEG): container finished" podID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerID="a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d" exitCode=0
Sep 08 18:40:31.417786 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.417774    2579 generic.go:356] "Generic (PLEG): container finished" podID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerID="6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366" exitCode=0
Sep 08 18:40:31.417969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.417816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerDied","Data":"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d"}
Sep 08 18:40:31.417969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.417848    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.417969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.417898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerDied","Data":"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366"}
Sep 08 18:40:31.417969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.417928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"322e1065-b5a3-4c06-ab36-1c01d2a53a01","Type":"ContainerDied","Data":"7cb98176c0cdeab115499c4a3a7d1aa290063fab28206086f479fdf26bb67994"}
Sep 08 18:40:31.418097 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.417972    2579 scope.go:117] "RemoveContainer" containerID="ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330"
Sep 08 18:40:31.425418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.425399    2579 scope.go:117] "RemoveContainer" containerID="a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d"
Sep 08 18:40:31.432299 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.432282    2579 scope.go:117] "RemoveContainer" containerID="180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df"
Sep 08 18:40:31.439028 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.438980    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:40:31.439097 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.439082    2579 scope.go:117] "RemoveContainer" containerID="6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366"
Sep 08 18:40:31.442678 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.442650    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:40:31.446228 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.446211    2579 scope.go:117] "RemoveContainer" containerID="dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229"
Sep 08 18:40:31.452869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.452855    2579 scope.go:117] "RemoveContainer" containerID="113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb"
Sep 08 18:40:31.459713 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.459698    2579 scope.go:117] "RemoveContainer" containerID="5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322"
Sep 08 18:40:31.466593 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.466572    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:40:31.466952 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.466936    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7218f5-d21b-4313-9bcf-9367a35bcd46" containerName="console"
Sep 08 18:40:31.466998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.466957    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7218f5-d21b-4313-9bcf-9367a35bcd46" containerName="console"
Sep 08 18:40:31.466998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.466971    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10" containerName="registry"
Sep 08 18:40:31.466998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.466980    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10" containerName="registry"
Sep 08 18:40:31.466998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.466991    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy-metric"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467000    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy-metric"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467012    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="alertmanager"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467021    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="alertmanager"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467032    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy-web"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467040    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy-web"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467058    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="prom-label-proxy"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467066    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="prom-label-proxy"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467082    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467090    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467109    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="init-config-reloader"
Sep 08 18:40:31.467124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467117    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="init-config-reloader"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467127    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="config-reloader"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467135    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="config-reloader"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467214    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467226    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="180ed34c-d8b9-4d11-a1b0-1d99770a7c10" containerName="registry"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467238    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy-metric"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467245    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7218f5-d21b-4313-9bcf-9367a35bcd46" containerName="console"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467249    2579 scope.go:117] "RemoveContainer" containerID="ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467252    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="config-reloader"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467330    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="alertmanager"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467361    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="kube-rbac-proxy-web"
Sep 08 18:40:31.467510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467372    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" containerName="prom-label-proxy"
Sep 08 18:40:31.467870 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:31.467541    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330\": container with ID starting with ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330 not found: ID does not exist" containerID="ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330"
Sep 08 18:40:31.467870 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467569    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330"} err="failed to get container status \"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330\": rpc error: code = NotFound desc = could not find container \"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330\": container with ID starting with ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330 not found: ID does not exist"
Sep 08 18:40:31.467870 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467592    2579 scope.go:117] "RemoveContainer" containerID="a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d"
Sep 08 18:40:31.468020 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:31.467869    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d\": container with ID starting with a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d not found: ID does not exist" containerID="a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d"
Sep 08 18:40:31.468020 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467903    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d"} err="failed to get container status \"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d\": rpc error: code = NotFound desc = could not find container \"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d\": container with ID starting with a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d not found: ID does not exist"
Sep 08 18:40:31.468020 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.467926    2579 scope.go:117] "RemoveContainer" containerID="180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df"
Sep 08 18:40:31.468246 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:31.468212    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df\": container with ID starting with 180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df not found: ID does not exist" containerID="180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df"
Sep 08 18:40:31.468293 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.468253    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df"} err="failed to get container status \"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df\": rpc error: code = NotFound desc = could not find container \"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df\": container with ID starting with 180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df not found: ID does not exist"
Sep 08 18:40:31.468293 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.468274    2579 scope.go:117] "RemoveContainer" containerID="6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366"
Sep 08 18:40:31.468560 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:31.468541    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366\": container with ID starting with 6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366 not found: ID does not exist" containerID="6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366"
Sep 08 18:40:31.468646 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.468569    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366"} err="failed to get container status \"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366\": rpc error: code = NotFound desc = could not find container \"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366\": container with ID starting with 6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366 not found: ID does not exist"
Sep 08 18:40:31.468646 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.468588    2579 scope.go:117] "RemoveContainer" containerID="dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229"
Sep 08 18:40:31.468850 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:31.468834    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229\": container with ID starting with dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229 not found: ID does not exist" containerID="dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229"
Sep 08 18:40:31.468909 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.468855    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229"} err="failed to get container status \"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229\": rpc error: code = NotFound desc = could not find container \"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229\": container with ID starting with dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229 not found: ID does not exist"
Sep 08 18:40:31.468909 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.468874    2579 scope.go:117] "RemoveContainer" containerID="113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb"
Sep 08 18:40:31.469143 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:31.469120    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb\": container with ID starting with 113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb not found: ID does not exist" containerID="113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb"
Sep 08 18:40:31.469203 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.469148    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb"} err="failed to get container status \"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb\": rpc error: code = NotFound desc = could not find container \"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb\": container with ID starting with 113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb not found: ID does not exist"
Sep 08 18:40:31.469203 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.469195    2579 scope.go:117] "RemoveContainer" containerID="5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322"
Sep 08 18:40:31.469507 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:31.469474    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322\": container with ID starting with 5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322 not found: ID does not exist" containerID="5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322"
Sep 08 18:40:31.469581 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.469513    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322"} err="failed to get container status \"5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322\": rpc error: code = NotFound desc = could not find container \"5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322\": container with ID starting with 5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322 not found: ID does not exist"
Sep 08 18:40:31.469581 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.469532    2579 scope.go:117] "RemoveContainer" containerID="ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330"
Sep 08 18:40:31.469806 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.469776    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330"} err="failed to get container status \"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330\": rpc error: code = NotFound desc = could not find container \"ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330\": container with ID starting with ee685ad6986b26cc4477053352e7035cb0edecce0194417cb3574068573c3330 not found: ID does not exist"
Sep 08 18:40:31.469806 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.469806    2579 scope.go:117] "RemoveContainer" containerID="a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d"
Sep 08 18:40:31.470182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.470153    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d"} err="failed to get container status \"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d\": rpc error: code = NotFound desc = could not find container \"a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d\": container with ID starting with a2b36dd232cf3b3b9bd5b9c188cbd045670042978df2466a18846bc22869df3d not found: ID does not exist"
Sep 08 18:40:31.470182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.470178    2579 scope.go:117] "RemoveContainer" containerID="180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df"
Sep 08 18:40:31.470565 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.470539    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df"} err="failed to get container status \"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df\": rpc error: code = NotFound desc = could not find container \"180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df\": container with ID starting with 180b4b66f7df7ab674a718a611fad1163d98d763db66a7f2bb7f16ea9e1126df not found: ID does not exist"
Sep 08 18:40:31.470565 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.470564    2579 scope.go:117] "RemoveContainer" containerID="6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366"
Sep 08 18:40:31.470891 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.470867    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366"} err="failed to get container status \"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366\": rpc error: code = NotFound desc = could not find container \"6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366\": container with ID starting with 6b400fbb71a6128d44e6ebdd5f39a1847a9e86fd00f4dfb0db53376164025366 not found: ID does not exist"
Sep 08 18:40:31.470962 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.470896    2579 scope.go:117] "RemoveContainer" containerID="dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229"
Sep 08 18:40:31.471149 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.471128    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229"} err="failed to get container status \"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229\": rpc error: code = NotFound desc = could not find container \"dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229\": container with ID starting with dca99c120c796485ba7b5c767954971da91cbce225b2be4455eabf80790f8229 not found: ID does not exist"
Sep 08 18:40:31.471214 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.471150    2579 scope.go:117] "RemoveContainer" containerID="113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb"
Sep 08 18:40:31.471538 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.471485    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb"} err="failed to get container status \"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb\": rpc error: code = NotFound desc = could not find container \"113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb\": container with ID starting with 113d3f972dd352957e0daea114c07e05f66450531a5a70ed0f53a22a614d6bfb not found: ID does not exist"
Sep 08 18:40:31.471538 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.471510    2579 scope.go:117] "RemoveContainer" containerID="5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322"
Sep 08 18:40:31.471796 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.471766    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322"} err="failed to get container status \"5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322\": rpc error: code = NotFound desc = could not find container \"5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322\": container with ID starting with 5bec9a3dac4a60e81a2b4bcd5d0fe7fae6bb64a425e81f221e33274830b57322 not found: ID does not exist"
Sep 08 18:40:31.474092 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.474076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.475909 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.475891    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 18:40:31.476001 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.475984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 18:40:31.476091 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.476075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 18:40:31.476329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.476311    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 18:40:31.476397 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.476330    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 18:40:31.476451 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.476438    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 18:40:31.476502 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.476457    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 18:40:31.476555 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.476460    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 18:40:31.476892 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.476865    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gkjbf\""
Sep 08 18:40:31.481890 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.481871    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 18:40:31.482926 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.482902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:40:31.597579 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597579 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27cmm\" (UniqueName: \"kubernetes.io/projected/14839ffa-94d7-4075-a2dd-85c53d4b5b28-kube-api-access-27cmm\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14839ffa-94d7-4075-a2dd-85c53d4b5b28-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-config-volume\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597996 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14839ffa-94d7-4075-a2dd-85c53d4b5b28-tls-assets\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597996 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14839ffa-94d7-4075-a2dd-85c53d4b5b28-config-out\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597996 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597996 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-web-config\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597996 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/14839ffa-94d7-4075-a2dd-85c53d4b5b28-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.597996 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.597988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14839ffa-94d7-4075-a2dd-85c53d4b5b28-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.698834 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.698756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14839ffa-94d7-4075-a2dd-85c53d4b5b28-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.698834 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.698789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.698834 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.698814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.698834 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.698833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-config-volume\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699147 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.698852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699147 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.698882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14839ffa-94d7-4075-a2dd-85c53d4b5b28-tls-assets\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699147 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.699046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14839ffa-94d7-4075-a2dd-85c53d4b5b28-config-out\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699147 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.699087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699147 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.699123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-web-config\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699438 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.699174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/14839ffa-94d7-4075-a2dd-85c53d4b5b28-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699438 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.699212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14839ffa-94d7-4075-a2dd-85c53d4b5b28-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699438 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.699249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.699438 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.699279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27cmm\" (UniqueName: \"kubernetes.io/projected/14839ffa-94d7-4075-a2dd-85c53d4b5b28-kube-api-access-27cmm\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.701123 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.701090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14839ffa-94d7-4075-a2dd-85c53d4b5b28-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.701892 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.701866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.701892 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.701881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/14839ffa-94d7-4075-a2dd-85c53d4b5b28-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.702033 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.701936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.702438 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.702412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-config-volume\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.702536 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.702266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14839ffa-94d7-4075-a2dd-85c53d4b5b28-tls-assets\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.702698 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.702676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14839ffa-94d7-4075-a2dd-85c53d4b5b28-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.704376 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.704130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.707724 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.704663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14839ffa-94d7-4075-a2dd-85c53d4b5b28-config-out\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.707724 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.704922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-web-config\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.707724 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.704967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.707724 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.705584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/14839ffa-94d7-4075-a2dd-85c53d4b5b28-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.707724 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.707111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27cmm\" (UniqueName: \"kubernetes.io/projected/14839ffa-94d7-4075-a2dd-85c53d4b5b28-kube-api-access-27cmm\") pod \"alertmanager-main-0\" (UID: \"14839ffa-94d7-4075-a2dd-85c53d4b5b28\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:31.784246 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:31.784216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:40:32.063855 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:32.063779    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:40:32.066990 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:40:32.066966    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14839ffa_94d7_4075_a2dd_85c53d4b5b28.slice/crio-1b02b3b1c07873186130e7675b5f22de5369cc1490f8facc9eb8d72048232d9a WatchSource:0}: Error finding container 1b02b3b1c07873186130e7675b5f22de5369cc1490f8facc9eb8d72048232d9a: Status 404 returned error can't find the container with id 1b02b3b1c07873186130e7675b5f22de5369cc1490f8facc9eb8d72048232d9a
Sep 08 18:40:32.162054 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:32.162027    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322e1065-b5a3-4c06-ab36-1c01d2a53a01" path="/var/lib/kubelet/pods/322e1065-b5a3-4c06-ab36-1c01d2a53a01/volumes"
Sep 08 18:40:32.422624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:32.422592    2579 generic.go:356] "Generic (PLEG): container finished" podID="14839ffa-94d7-4075-a2dd-85c53d4b5b28" containerID="754a581dd7ab083b3115ca8a42cd7db7957811c9f1466a9243d7a3036542f3f3" exitCode=0
Sep 08 18:40:32.422828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:32.422687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"14839ffa-94d7-4075-a2dd-85c53d4b5b28","Type":"ContainerDied","Data":"754a581dd7ab083b3115ca8a42cd7db7957811c9f1466a9243d7a3036542f3f3"}
Sep 08 18:40:32.422828 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:32.422762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"14839ffa-94d7-4075-a2dd-85c53d4b5b28","Type":"ContainerStarted","Data":"1b02b3b1c07873186130e7675b5f22de5369cc1490f8facc9eb8d72048232d9a"}
Sep 08 18:40:33.433762 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.433727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"14839ffa-94d7-4075-a2dd-85c53d4b5b28","Type":"ContainerStarted","Data":"e2220a183380bd6c9c87594db24b6dec03e8145bef4784a59e08fa0f06eed4cf"}
Sep 08 18:40:33.434243 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.433806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"14839ffa-94d7-4075-a2dd-85c53d4b5b28","Type":"ContainerStarted","Data":"a5f460ce72cc2251cd88305712e1856a0ee417c86b813ef4daad0f9dea9235bf"}
Sep 08 18:40:33.434243 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.433848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"14839ffa-94d7-4075-a2dd-85c53d4b5b28","Type":"ContainerStarted","Data":"2aad462cd26f49107f0ffe83952e1c4d737f5b9bb3a827c4e7ec546bd55984ae"}
Sep 08 18:40:33.434243 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.433894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"14839ffa-94d7-4075-a2dd-85c53d4b5b28","Type":"ContainerStarted","Data":"19aa706c20486f87281f9ef733992e1a306587ad185acb46d1d660ed3c3d0c56"}
Sep 08 18:40:33.434243 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.433947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"14839ffa-94d7-4075-a2dd-85c53d4b5b28","Type":"ContainerStarted","Data":"959b83b5a48fe1fc847d4786f61d0785a04c66294c55d22b84267b10333790b1"}
Sep 08 18:40:33.434243 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.433987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"14839ffa-94d7-4075-a2dd-85c53d4b5b28","Type":"ContainerStarted","Data":"5ebcf89192580b43f74b1290e02b72ebaee1224b64795810f27b6b38f8127f75"}
Sep 08 18:40:33.457152 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.457088    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.442912743 podStartE2EDuration="2.45706763s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.154887ms" isStatefulPod=false podCreationTimestamp="2026-09-08 18:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:33.45561424 +0000 UTC m=+393.809707574" watchObservedRunningTime="2026-09-08 18:40:33.45706763 +0000 UTC m=+393.811160964"
Sep 08 18:40:33.816163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.816070    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-84455f89f4-9j82w"]
Sep 08 18:40:33.820087 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.820061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:33.822188 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.822152    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 08 18:40:33.822321 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.822263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 08 18:40:33.822404 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.822264    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 08 18:40:33.822404 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.822329    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 08 18:40:33.822516 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.822428    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 08 18:40:33.822516 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.822459    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-p7wct\""
Sep 08 18:40:33.829002 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.828978    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 08 18:40:33.832904 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.832878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84455f89f4-9j82w"]
Sep 08 18:40:33.918801 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.918762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-metrics-client-ca\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:33.918990 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.918825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-secret-telemeter-client\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:33.918990 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.918854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfqwn\" (UniqueName: \"kubernetes.io/projected/349ece67-7975-47ee-bc5c-7af76697caed-kube-api-access-nfqwn\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:33.918990 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.918892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:33.919146 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.919007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-federate-client-tls\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:33.919146 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.919106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-telemeter-client-tls\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:33.919242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.919193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-serving-certs-ca-bundle\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:33.919242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:33.919224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.020529 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.020487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-serving-certs-ca-bundle\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.020688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.020543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.020688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.020594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-metrics-client-ca\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.020822 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.020742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-secret-telemeter-client\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.020822 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.020785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfqwn\" (UniqueName: \"kubernetes.io/projected/349ece67-7975-47ee-bc5c-7af76697caed-kube-api-access-nfqwn\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.020995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.020972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.021082 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.021048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-federate-client-tls\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.021135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.021097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-telemeter-client-tls\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.021541 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.021328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-metrics-client-ca\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.021541 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.021465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-serving-certs-ca-bundle\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.021541 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.021527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349ece67-7975-47ee-bc5c-7af76697caed-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.023604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.023576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.023782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.023761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-secret-telemeter-client\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.024567 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.024050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-federate-client-tls\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.024567 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.024097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/349ece67-7975-47ee-bc5c-7af76697caed-telemeter-client-tls\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.027715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.027693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfqwn\" (UniqueName: \"kubernetes.io/projected/349ece67-7975-47ee-bc5c-7af76697caed-kube-api-access-nfqwn\") pod \"telemeter-client-84455f89f4-9j82w\" (UID: \"349ece67-7975-47ee-bc5c-7af76697caed\") " pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.088619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.088578    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:40:34.089015 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.088961    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="prometheus" containerID="cri-o://652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7" gracePeriod=600
Sep 08 18:40:34.089015 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.088974    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy" containerID="cri-o://5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6" gracePeriod=600
Sep 08 18:40:34.089015 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.088991    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="thanos-sidecar" containerID="cri-o://f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da" gracePeriod=600
Sep 08 18:40:34.089224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.089010    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy-web" containerID="cri-o://562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373" gracePeriod=600
Sep 08 18:40:34.089224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.089031    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="config-reloader" containerID="cri-o://73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e" gracePeriod=600
Sep 08 18:40:34.089224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.088991    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b" gracePeriod=600
Sep 08 18:40:34.130051 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.130023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w"
Sep 08 18:40:34.355322 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.355292    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.423054 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.423022    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84455f89f4-9j82w"]
Sep 08 18:40:34.424547 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424525    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-trusted-ca-bundle\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.424668 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-tls-assets\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.424668 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-thanos-prometheus-http-client-file\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.424668 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-grpc-tls\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.424883 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424675    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-metrics-client-certs\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.424883 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424708    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-2\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.424883 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424825    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.424883 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-1\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424896    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-0\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:34.425044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.424938    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-db\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425003    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-tls\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425030    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-kubelet-serving-ca-bundle\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425087    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-serving-certs-ca-bundle\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425120    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-metrics-client-ca\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config-out\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-kube-rbac-proxy\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425219    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-962gq\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-kube-api-access-962gq\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-web-config\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425635 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\" (UID: \"16c8b639-1054-43ca-a70d-bbf094b8e5dd\") "
Sep 08 18:40:34.425877 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.425856    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.426058 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.426032    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:34.430182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.429093    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:34.430182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.429539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:40:34.430182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.429745    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:34.430182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.429800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.430182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.430157    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:34.430182 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.430174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.430568 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.430225    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:34.430568 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.430266    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.430568 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.430419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.430568 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.430520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:34.431184 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.431159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:40:34.431507 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.431354    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.431649 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.431431    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-kube-api-access-962gq" (OuterVolumeSpecName: "kube-api-access-962gq") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "kube-api-access-962gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:40:34.432485 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.431492    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config-out" (OuterVolumeSpecName: "config-out") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:40:34.432485 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.431791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.432613 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.432253    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.433254 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.433233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config" (OuterVolumeSpecName: "config") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.442174 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442146    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-web-config" (OuterVolumeSpecName: "web-config") pod "16c8b639-1054-43ca-a70d-bbf094b8e5dd" (UID: "16c8b639-1054-43ca-a70d-bbf094b8e5dd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:34.442701 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442676    2579 generic.go:356] "Generic (PLEG): container finished" podID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b" exitCode=0
Sep 08 18:40:34.442760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442701    2579 generic.go:356] "Generic (PLEG): container finished" podID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6" exitCode=0
Sep 08 18:40:34.442760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442711    2579 generic.go:356] "Generic (PLEG): container finished" podID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373" exitCode=0
Sep 08 18:40:34.442760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442719    2579 generic.go:356] "Generic (PLEG): container finished" podID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da" exitCode=0
Sep 08 18:40:34.442760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442727    2579 generic.go:356] "Generic (PLEG): container finished" podID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e" exitCode=0
Sep 08 18:40:34.442760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442735    2579 generic.go:356] "Generic (PLEG): container finished" podID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7" exitCode=0
Sep 08 18:40:34.442992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442756    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.442992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerDied","Data":"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"}
Sep 08 18:40:34.442992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerDied","Data":"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"}
Sep 08 18:40:34.442992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerDied","Data":"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"}
Sep 08 18:40:34.442992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442911    2579 scope.go:117] "RemoveContainer" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"
Sep 08 18:40:34.442992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerDied","Data":"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"}
Sep 08 18:40:34.442992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.442977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerDied","Data":"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"}
Sep 08 18:40:34.443321 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.443017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerDied","Data":"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"}
Sep 08 18:40:34.443321 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.443061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16c8b639-1054-43ca-a70d-bbf094b8e5dd","Type":"ContainerDied","Data":"ee150a989223f950db4e51db4011533399843664c28b00542f25d6dc0f5a4c0b"}
Sep 08 18:40:34.444376 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.444329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w" event={"ID":"349ece67-7975-47ee-bc5c-7af76697caed","Type":"ContainerStarted","Data":"84cc9d83012e8bba016b9f8f00ce64ad7ddf15573c2d0ca6cf0529b325e7b5ae"}
Sep 08 18:40:34.452599 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.452580    2579 scope.go:117] "RemoveContainer" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"
Sep 08 18:40:34.460563 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.460546    2579 scope.go:117] "RemoveContainer" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"
Sep 08 18:40:34.467220 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.467199    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:40:34.467593 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.467576    2579 scope.go:117] "RemoveContainer" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"
Sep 08 18:40:34.473569 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.473547    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:40:34.474646 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.474630    2579 scope.go:117] "RemoveContainer" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"
Sep 08 18:40:34.480911 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.480894    2579 scope.go:117] "RemoveContainer" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"
Sep 08 18:40:34.487896 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.487881    2579 scope.go:117] "RemoveContainer" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"
Sep 08 18:40:34.494255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.494241    2579 scope.go:117] "RemoveContainer" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"
Sep 08 18:40:34.494572 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:34.494551    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": container with ID starting with e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b not found: ID does not exist" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"
Sep 08 18:40:34.494640 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.494576    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"} err="failed to get container status \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": rpc error: code = NotFound desc = could not find container \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": container with ID starting with e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b not found: ID does not exist"
Sep 08 18:40:34.494640 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.494594    2579 scope.go:117] "RemoveContainer" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"
Sep 08 18:40:34.494854 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:34.494837    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": container with ID starting with 5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6 not found: ID does not exist" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"
Sep 08 18:40:34.494901 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.494860    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"} err="failed to get container status \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": rpc error: code = NotFound desc = could not find container \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": container with ID starting with 5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6 not found: ID does not exist"
Sep 08 18:40:34.494901 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.494875    2579 scope.go:117] "RemoveContainer" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"
Sep 08 18:40:34.495139 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:34.495124    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": container with ID starting with 562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373 not found: ID does not exist" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"
Sep 08 18:40:34.495197 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.495151    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"} err="failed to get container status \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": rpc error: code = NotFound desc = could not find container \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": container with ID starting with 562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373 not found: ID does not exist"
Sep 08 18:40:34.495197 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.495163    2579 scope.go:117] "RemoveContainer" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"
Sep 08 18:40:34.495426 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:34.495410    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": container with ID starting with f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da not found: ID does not exist" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"
Sep 08 18:40:34.495471 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.495429    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"} err="failed to get container status \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": rpc error: code = NotFound desc = could not find container \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": container with ID starting with f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da not found: ID does not exist"
Sep 08 18:40:34.495471 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.495444    2579 scope.go:117] "RemoveContainer" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"
Sep 08 18:40:34.495689 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:34.495671    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": container with ID starting with 73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e not found: ID does not exist" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"
Sep 08 18:40:34.495737 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.495692    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"} err="failed to get container status \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": rpc error: code = NotFound desc = could not find container \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": container with ID starting with 73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e not found: ID does not exist"
Sep 08 18:40:34.495737 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.495706    2579 scope.go:117] "RemoveContainer" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"
Sep 08 18:40:34.495935 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:34.495916    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": container with ID starting with 652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7 not found: ID does not exist" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"
Sep 08 18:40:34.495994 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.495937    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"} err="failed to get container status \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": rpc error: code = NotFound desc = could not find container \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": container with ID starting with 652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7 not found: ID does not exist"
Sep 08 18:40:34.495994 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.495949    2579 scope.go:117] "RemoveContainer" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"
Sep 08 18:40:34.496307 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:40:34.496247    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": container with ID starting with 04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766 not found: ID does not exist" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"
Sep 08 18:40:34.496430 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.496317    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"} err="failed to get container status \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": rpc error: code = NotFound desc = could not find container \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": container with ID starting with 04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766 not found: ID does not exist"
Sep 08 18:40:34.496430 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.496361    2579 scope.go:117] "RemoveContainer" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"
Sep 08 18:40:34.496684 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.496660    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"} err="failed to get container status \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": rpc error: code = NotFound desc = could not find container \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": container with ID starting with e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b not found: ID does not exist"
Sep 08 18:40:34.496684 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.496684    2579 scope.go:117] "RemoveContainer" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"
Sep 08 18:40:34.497167 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497138    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"} err="failed to get container status \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": rpc error: code = NotFound desc = could not find container \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": container with ID starting with 5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6 not found: ID does not exist"
Sep 08 18:40:34.497167 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497161    2579 scope.go:117] "RemoveContainer" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"
Sep 08 18:40:34.497450 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497424    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"} err="failed to get container status \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": rpc error: code = NotFound desc = could not find container \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": container with ID starting with 562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373 not found: ID does not exist"
Sep 08 18:40:34.497450 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497449    2579 scope.go:117] "RemoveContainer" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"
Sep 08 18:40:34.497687 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497671    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"} err="failed to get container status \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": rpc error: code = NotFound desc = could not find container \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": container with ID starting with f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da not found: ID does not exist"
Sep 08 18:40:34.497735 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497688    2579 scope.go:117] "RemoveContainer" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"
Sep 08 18:40:34.497955 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497938    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"} err="failed to get container status \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": rpc error: code = NotFound desc = could not find container \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": container with ID starting with 73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e not found: ID does not exist"
Sep 08 18:40:34.498000 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497952    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:40:34.498039 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.497956    2579 scope.go:117] "RemoveContainer" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"
Sep 08 18:40:34.498251 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498230    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="thanos-sidecar"
Sep 08 18:40:34.498251 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498245    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="thanos-sidecar"
Sep 08 18:40:34.498329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498254    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy-web"
Sep 08 18:40:34.498329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498260    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy-web"
Sep 08 18:40:34.498329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498257    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"} err="failed to get container status \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": rpc error: code = NotFound desc = could not find container \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": container with ID starting with 652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7 not found: ID does not exist"
Sep 08 18:40:34.498329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498277    2579 scope.go:117] "RemoveContainer" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"
Sep 08 18:40:34.498329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498269    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy-thanos"
Sep 08 18:40:34.498329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498329    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy-thanos"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498360    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="prometheus"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498370    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="prometheus"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498386    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="init-config-reloader"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498395    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="init-config-reloader"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498404    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="config-reloader"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498413    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="config-reloader"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498422    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498430    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498536    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="prometheus"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498546    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy-thanos"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498553    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498559    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="config-reloader"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498565    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="thanos-sidecar"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498558    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"} err="failed to get container status \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": rpc error: code = NotFound desc = could not find container \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": container with ID starting with 04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766 not found: ID does not exist"
Sep 08 18:40:34.498604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498588    2579 scope.go:117] "RemoveContainer" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"
Sep 08 18:40:34.499283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498571    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" containerName="kube-rbac-proxy-web"
Sep 08 18:40:34.499283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498834    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"} err="failed to get container status \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": rpc error: code = NotFound desc = could not find container \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": container with ID starting with e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b not found: ID does not exist"
Sep 08 18:40:34.499283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.498850    2579 scope.go:117] "RemoveContainer" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"
Sep 08 18:40:34.499283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.499079    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"} err="failed to get container status \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": rpc error: code = NotFound desc = could not find container \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": container with ID starting with 5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6 not found: ID does not exist"
Sep 08 18:40:34.499283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.499095    2579 scope.go:117] "RemoveContainer" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"
Sep 08 18:40:34.499546 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.499302    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"} err="failed to get container status \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": rpc error: code = NotFound desc = could not find container \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": container with ID starting with 562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373 not found: ID does not exist"
Sep 08 18:40:34.499546 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.499316    2579 scope.go:117] "RemoveContainer" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"
Sep 08 18:40:34.499638 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.499615    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"} err="failed to get container status \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": rpc error: code = NotFound desc = could not find container \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": container with ID starting with f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da not found: ID does not exist"
Sep 08 18:40:34.499638 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.499633    2579 scope.go:117] "RemoveContainer" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"
Sep 08 18:40:34.499847 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.499824    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"} err="failed to get container status \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": rpc error: code = NotFound desc = could not find container \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": container with ID starting with 73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e not found: ID does not exist"
Sep 08 18:40:34.499847 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.499846    2579 scope.go:117] "RemoveContainer" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"
Sep 08 18:40:34.500098 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.500080    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"} err="failed to get container status \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": rpc error: code = NotFound desc = could not find container \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": container with ID starting with 652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7 not found: ID does not exist"
Sep 08 18:40:34.500162 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.500099    2579 scope.go:117] "RemoveContainer" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"
Sep 08 18:40:34.500365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.500315    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"} err="failed to get container status \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": rpc error: code = NotFound desc = could not find container \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": container with ID starting with 04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766 not found: ID does not exist"
Sep 08 18:40:34.500365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.500357    2579 scope.go:117] "RemoveContainer" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"
Sep 08 18:40:34.500572 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.500555    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"} err="failed to get container status \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": rpc error: code = NotFound desc = could not find container \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": container with ID starting with e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b not found: ID does not exist"
Sep 08 18:40:34.500611 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.500574    2579 scope.go:117] "RemoveContainer" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"
Sep 08 18:40:34.500821 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.500805    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"} err="failed to get container status \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": rpc error: code = NotFound desc = could not find container \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": container with ID starting with 5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6 not found: ID does not exist"
Sep 08 18:40:34.500861 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.500821    2579 scope.go:117] "RemoveContainer" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"
Sep 08 18:40:34.501050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501031    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"} err="failed to get container status \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": rpc error: code = NotFound desc = could not find container \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": container with ID starting with 562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373 not found: ID does not exist"
Sep 08 18:40:34.501090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501053    2579 scope.go:117] "RemoveContainer" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"
Sep 08 18:40:34.501278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501262    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"} err="failed to get container status \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": rpc error: code = NotFound desc = could not find container \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": container with ID starting with f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da not found: ID does not exist"
Sep 08 18:40:34.501321 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501279    2579 scope.go:117] "RemoveContainer" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"
Sep 08 18:40:34.501530 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501511    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"} err="failed to get container status \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": rpc error: code = NotFound desc = could not find container \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": container with ID starting with 73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e not found: ID does not exist"
Sep 08 18:40:34.501571 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501532    2579 scope.go:117] "RemoveContainer" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"
Sep 08 18:40:34.501779 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501761    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"} err="failed to get container status \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": rpc error: code = NotFound desc = could not find container \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": container with ID starting with 652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7 not found: ID does not exist"
Sep 08 18:40:34.501825 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501781    2579 scope.go:117] "RemoveContainer" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"
Sep 08 18:40:34.502010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.501989    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"} err="failed to get container status \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": rpc error: code = NotFound desc = could not find container \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": container with ID starting with 04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766 not found: ID does not exist"
Sep 08 18:40:34.502064 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.502012    2579 scope.go:117] "RemoveContainer" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"
Sep 08 18:40:34.502302 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.502281    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"} err="failed to get container status \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": rpc error: code = NotFound desc = could not find container \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": container with ID starting with e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b not found: ID does not exist"
Sep 08 18:40:34.502302 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.502299    2579 scope.go:117] "RemoveContainer" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"
Sep 08 18:40:34.502607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.502582    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"} err="failed to get container status \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": rpc error: code = NotFound desc = could not find container \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": container with ID starting with 5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6 not found: ID does not exist"
Sep 08 18:40:34.502607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.502604    2579 scope.go:117] "RemoveContainer" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"
Sep 08 18:40:34.502879 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.502858    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"} err="failed to get container status \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": rpc error: code = NotFound desc = could not find container \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": container with ID starting with 562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373 not found: ID does not exist"
Sep 08 18:40:34.502879 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.502878    2579 scope.go:117] "RemoveContainer" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"
Sep 08 18:40:34.503224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.503172    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"} err="failed to get container status \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": rpc error: code = NotFound desc = could not find container \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": container with ID starting with f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da not found: ID does not exist"
Sep 08 18:40:34.503224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.503198    2579 scope.go:117] "RemoveContainer" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"
Sep 08 18:40:34.503495 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.503470    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"} err="failed to get container status \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": rpc error: code = NotFound desc = could not find container \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": container with ID starting with 73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e not found: ID does not exist"
Sep 08 18:40:34.503583 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.503497    2579 scope.go:117] "RemoveContainer" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"
Sep 08 18:40:34.503766 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.503739    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"} err="failed to get container status \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": rpc error: code = NotFound desc = could not find container \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": container with ID starting with 652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7 not found: ID does not exist"
Sep 08 18:40:34.503766 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.503767    2579 scope.go:117] "RemoveContainer" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"
Sep 08 18:40:34.504041 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.503997    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"} err="failed to get container status \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": rpc error: code = NotFound desc = could not find container \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": container with ID starting with 04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766 not found: ID does not exist"
Sep 08 18:40:34.504041 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.504019    2579 scope.go:117] "RemoveContainer" containerID="e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"
Sep 08 18:40:34.504261 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.504240    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b"} err="failed to get container status \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": rpc error: code = NotFound desc = could not find container \"e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b\": container with ID starting with e125ee804a5b92d08423eeb58eb7bb50354c623920e36d54f2926b9822ff210b not found: ID does not exist"
Sep 08 18:40:34.504370 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.504262    2579 scope.go:117] "RemoveContainer" containerID="5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"
Sep 08 18:40:34.504599 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.504520    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6"} err="failed to get container status \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": rpc error: code = NotFound desc = could not find container \"5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6\": container with ID starting with 5b7dba14dc0bdefa1d7880170eca5630a638a973ea4090a810b58438d9e95fd6 not found: ID does not exist"
Sep 08 18:40:34.504599 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.504548    2579 scope.go:117] "RemoveContainer" containerID="562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"
Sep 08 18:40:34.504829 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.504808    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373"} err="failed to get container status \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": rpc error: code = NotFound desc = could not find container \"562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373\": container with ID starting with 562ecdb5d9325d8f7ef1997eb0e6cfbef85bf2d95571809c639e08c8bbfef373 not found: ID does not exist"
Sep 08 18:40:34.504898 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.504831    2579 scope.go:117] "RemoveContainer" containerID="f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"
Sep 08 18:40:34.505165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.505105    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da"} err="failed to get container status \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": rpc error: code = NotFound desc = could not find container \"f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da\": container with ID starting with f91cc23b761c074e972310b4dfba7a1ae9f57eba0965043d4854e1b30aa2c5da not found: ID does not exist"
Sep 08 18:40:34.505165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.505123    2579 scope.go:117] "RemoveContainer" containerID="73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"
Sep 08 18:40:34.505393 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.505365    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e"} err="failed to get container status \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": rpc error: code = NotFound desc = could not find container \"73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e\": container with ID starting with 73cb1bd9c71ec2a956b29ddb0c5f1ee19d3277bb51503642ca8adb13f28f413e not found: ID does not exist"
Sep 08 18:40:34.505393 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.505386    2579 scope.go:117] "RemoveContainer" containerID="652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"
Sep 08 18:40:34.505535 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.505517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.505654 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.505622    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7"} err="failed to get container status \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": rpc error: code = NotFound desc = could not find container \"652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7\": container with ID starting with 652c53609fa16acfcb970357144c88671d5b152267b5b978cddc218534e65af7 not found: ID does not exist"
Sep 08 18:40:34.505654 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.505643    2579 scope.go:117] "RemoveContainer" containerID="04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"
Sep 08 18:40:34.506165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.506144    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766"} err="failed to get container status \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": rpc error: code = NotFound desc = could not find container \"04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766\": container with ID starting with 04d226bd760c8d048cf8f9d710d996dbe5a14f56b587389a03d3e4bf0c518766 not found: ID does not exist"
Sep 08 18:40:34.507627 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.507609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 18:40:34.507830 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.507818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 18:40:34.507999 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.507982    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 18:40:34.508052 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508004    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 18:40:34.508099 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508087    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 18:40:34.508141 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508106    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 18:40:34.508284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508267    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 18:40:34.508366 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508322    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 18:40:34.508418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fl8m9nac3ktn9\""
Sep 08 18:40:34.508418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 18:40:34.508418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508398    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 18:40:34.508686 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508670    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c87rq\""
Sep 08 18:40:34.508772 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.508756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 18:40:34.509437 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.509422    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 18:40:34.509856 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.509843    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 18:40:34.511507 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.511487    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 18:40:34.514195 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.514177    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 18:40:34.521495 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.521477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:40:34.526683 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.526662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.526759 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.526699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2592e0d-d3ff-499f-ac51-8321e602937f-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.526759 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.526725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.526898 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.526873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.526935 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.526916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527013 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.526995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527056 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527100 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527100 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwtqx\" (UniqueName: \"kubernetes.io/projected/d2592e0d-d3ff-499f-ac51-8321e602937f-kube-api-access-dwtqx\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527201 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-config\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527201 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527201 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527332 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527332 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2592e0d-d3ff-499f-ac51-8321e602937f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527535 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527592 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527815 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.527915 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527899    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-grpc-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.527961 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527918    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-metrics-client-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.527961 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527931    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.527961 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527942    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527961    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527974    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527983    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-prometheus-k8s-db\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.527992    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528000    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528016    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528029    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528038    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16c8b639-1054-43ca-a70d-bbf094b8e5dd-configmap-metrics-client-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528047    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16c8b639-1054-43ca-a70d-bbf094b8e5dd-config-out\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528056    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-kube-rbac-proxy\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528066    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-962gq\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-kube-api-access-962gq\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528074    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-web-config\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528084    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528093    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16c8b639-1054-43ca-a70d-bbf094b8e5dd-tls-assets\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.528365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.528102    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16c8b639-1054-43ca-a70d-bbf094b8e5dd-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:40:34.629388 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629388 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2592e0d-d3ff-499f-ac51-8321e602937f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629388 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629645 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629645 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629645 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629645 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629645 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2592e0d-d3ff-499f-ac51-8321e602937f-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629645 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629645 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwtqx\" (UniqueName: \"kubernetes.io/projected/d2592e0d-d3ff-499f-ac51-8321e602937f-kube-api-access-dwtqx\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-config\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.629988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.629918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.630480 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.630357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.630544 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.630498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.631259 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.630910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.631259 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.630955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.631444 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.631374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.632530 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.632501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.632617 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.632559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.632694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.632675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.633590 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.633070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.633590 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.633249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2592e0d-d3ff-499f-ac51-8321e602937f-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.633590 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.633327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.634074 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.634037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.634241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.634218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2592e0d-d3ff-499f-ac51-8321e602937f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.634324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.634269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.634324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.634290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.634461 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.634335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2592e0d-d3ff-499f-ac51-8321e602937f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.634960 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.634943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.635451 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.635429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-config\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.635712 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.635696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2592e0d-d3ff-499f-ac51-8321e602937f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.638491 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.638475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwtqx\" (UniqueName: \"kubernetes.io/projected/d2592e0d-d3ff-499f-ac51-8321e602937f-kube-api-access-dwtqx\") pod \"prometheus-k8s-0\" (UID: \"d2592e0d-d3ff-499f-ac51-8321e602937f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:34.816486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:34.816446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:35.097285 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:35.097233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:40:35.098399 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:40:35.098372    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2592e0d_d3ff_499f_ac51_8321e602937f.slice/crio-5e919655eccf354372e680f8965bfe89286cd37d492803e8445f50c9ac52c376 WatchSource:0}: Error finding container 5e919655eccf354372e680f8965bfe89286cd37d492803e8445f50c9ac52c376: Status 404 returned error can't find the container with id 5e919655eccf354372e680f8965bfe89286cd37d492803e8445f50c9ac52c376
Sep 08 18:40:35.451880 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:35.451834    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2592e0d-d3ff-499f-ac51-8321e602937f" containerID="a37c79ce5ee3877576a7f2c9f96d76c027e4774dd53f1c35c851ff810dd4adf5" exitCode=0
Sep 08 18:40:35.452336 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:35.451924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2592e0d-d3ff-499f-ac51-8321e602937f","Type":"ContainerDied","Data":"a37c79ce5ee3877576a7f2c9f96d76c027e4774dd53f1c35c851ff810dd4adf5"}
Sep 08 18:40:35.452336 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:35.452013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2592e0d-d3ff-499f-ac51-8321e602937f","Type":"ContainerStarted","Data":"5e919655eccf354372e680f8965bfe89286cd37d492803e8445f50c9ac52c376"}
Sep 08 18:40:36.165447 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.165412    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16c8b639-1054-43ca-a70d-bbf094b8e5dd" path="/var/lib/kubelet/pods/16c8b639-1054-43ca-a70d-bbf094b8e5dd/volumes"
Sep 08 18:40:36.459864 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.459838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w" event={"ID":"349ece67-7975-47ee-bc5c-7af76697caed","Type":"ContainerStarted","Data":"1fcd7d019954299715c3fe309a32380df58106766b2e1daea11b0490aaaa48a4"}
Sep 08 18:40:36.460209 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.459892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w" event={"ID":"349ece67-7975-47ee-bc5c-7af76697caed","Type":"ContainerStarted","Data":"0f764233830303097ef3e8a06d3aa145f2c65792498ed9a27b28628c894b2942"}
Sep 08 18:40:36.466048 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.466020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2592e0d-d3ff-499f-ac51-8321e602937f","Type":"ContainerStarted","Data":"238c70338689e41e23236327579cbdb258e063277d75ab54fe0abdaeedf0ef1d"}
Sep 08 18:40:36.466163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.466075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2592e0d-d3ff-499f-ac51-8321e602937f","Type":"ContainerStarted","Data":"26867d7569424b4702969bacf15282c5a87ee7ebaa5780e9767af5fb70bf7bfe"}
Sep 08 18:40:36.466163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.466099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2592e0d-d3ff-499f-ac51-8321e602937f","Type":"ContainerStarted","Data":"0a8135ecda463bcb85ba947a538fd20dbbe09ae9ee5eab90d39a4d94d479bfbb"}
Sep 08 18:40:36.466163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.466125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2592e0d-d3ff-499f-ac51-8321e602937f","Type":"ContainerStarted","Data":"4937932f875231c7a99536b4a930f6f9209b5293676fa8e9d65d3bb321abac8a"}
Sep 08 18:40:36.466163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.466149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2592e0d-d3ff-499f-ac51-8321e602937f","Type":"ContainerStarted","Data":"db1e9837dbcc02b89ddfb02629e22e8370a66a614cb193c7914f81d169bdf1de"}
Sep 08 18:40:36.466297 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.466170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2592e0d-d3ff-499f-ac51-8321e602937f","Type":"ContainerStarted","Data":"55151583876dc29c3e7e593ed5c16d0ac02875dd7025d9d735d27598b9934b25"}
Sep 08 18:40:36.491470 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:36.490873    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.466777888 podStartE2EDuration="2.490854197s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.076309ms" isStatefulPod=false podCreationTimestamp="2026-09-08 18:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:36.488887044 +0000 UTC m=+396.842980388" watchObservedRunningTime="2026-09-08 18:40:36.490854197 +0000 UTC m=+396.844947531"
Sep 08 18:40:37.472580 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:37.472546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w" event={"ID":"349ece67-7975-47ee-bc5c-7af76697caed","Type":"ContainerStarted","Data":"61c95a310c82b95355d1e16f8e65cca6310f9e63d049733a868b0d56d9e615ef"}
Sep 08 18:40:37.491113 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:37.491057    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-84455f89f4-9j82w" podStartSLOduration=2.620193213 podStartE2EDuration="4.491038379s" totalImagesPullingTime="1.870845166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:37.489418352 +0000 UTC m=+397.843511684" watchObservedRunningTime="2026-09-08 18:40:37.491038379 +0000 UTC m=+397.845131715"
Sep 08 18:40:37.628681 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:37.628651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:37.783554 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:37.783477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:39.817820 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:39.817780    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:40:41.084531 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:41.084494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:41.135361 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:41.135317    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:48.155799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:48.155767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:48.306384 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:48.306335    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:48.359153 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:48.359124    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:40:55.149164 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:40:55.149126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:27.044179 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:27.044099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:27.096212 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:27.096180    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:34.817498 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:34.817444    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:41:34.833014 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:34.832987    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:41:35.714879 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:35.714854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:41:37.690690 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:37.690656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:43.946197 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:43.946161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:43.998992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:43.998959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:49.835625 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:49.835595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:49.887528 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:49.887495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:55.919164 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:55.919124    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:41:55.967381 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:41:55.967318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:42:00.446045 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:42:00.445993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:42:00.502714 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:42:00.502676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:42:08.638008 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:42:08.637973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:42:08.802432 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:42:08.802401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:42:08.867534 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:42:08.867500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:05.921875 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:05.921841    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:06.054032 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:06.053994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:31.298607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:31.298567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:38.462299 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:38.462265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:39.238483 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:39.238451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:42.067755 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:42.067717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:42.167474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:42.167446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:44.870475 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:44.870442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:44.972739 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:44.972700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:45.377030 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:45.376998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:45.482541 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:45.482507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:46.887826 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:46.887796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:47.263639 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:47.263561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:48.038712 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:48.038680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:48.138757 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:48.138724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:55.794570 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:55.794534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:56.465962 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:56.465926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:56.563518 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:56.563453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:57.102096 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:57.102058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:43:57.182533 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:43:57.182496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:00.074751 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:00.074724    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:44:00.077039 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:00.077018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:44:01.785365 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:01.785313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:01.886544 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:01.886511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:03.621677 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:03.621644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:11.819532 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.819490    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:11.881151 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.881115    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"]
Sep 08 18:44:11.884461 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.884438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:11.887085 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.887050    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 18:44:11.887085 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.887067    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:44:11.887379 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.887058    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 18:44:11.887379 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.887109    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 18:44:11.887379 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.887128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6hblb\""
Sep 08 18:44:11.887379 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.887094    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 18:44:11.893111 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.893089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"]
Sep 08 18:44:11.954495 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.954450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55216b79-6258-4fc1-8568-4d7f6559d00c-cert\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:11.954672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.954515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8gh\" (UniqueName: \"kubernetes.io/projected/55216b79-6258-4fc1-8568-4d7f6559d00c-kube-api-access-pg8gh\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:11.954672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.954594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55216b79-6258-4fc1-8568-4d7f6559d00c-metrics-cert\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:11.954672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:11.954641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55216b79-6258-4fc1-8568-4d7f6559d00c-manager-config\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.055473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.055432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55216b79-6258-4fc1-8568-4d7f6559d00c-manager-config\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.055625 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.055489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55216b79-6258-4fc1-8568-4d7f6559d00c-cert\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.055625 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.055520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8gh\" (UniqueName: \"kubernetes.io/projected/55216b79-6258-4fc1-8568-4d7f6559d00c-kube-api-access-pg8gh\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.055625 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.055549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55216b79-6258-4fc1-8568-4d7f6559d00c-metrics-cert\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.056130 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.056108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55216b79-6258-4fc1-8568-4d7f6559d00c-manager-config\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.057928 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.057904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55216b79-6258-4fc1-8568-4d7f6559d00c-cert\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.058013 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.057947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55216b79-6258-4fc1-8568-4d7f6559d00c-metrics-cert\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.063734 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.063705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8gh\" (UniqueName: \"kubernetes.io/projected/55216b79-6258-4fc1-8568-4d7f6559d00c-kube-api-access-pg8gh\") pod \"lws-controller-manager-68bb56fbd7-f5m5j\" (UID: \"55216b79-6258-4fc1-8568-4d7f6559d00c\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.194645 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.194615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:12.462504 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.462429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"]
Sep 08 18:44:12.465724 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:44:12.465698    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55216b79_6258_4fc1_8568_4d7f6559d00c.slice/crio-ab6d8be7e1afb6e3ae6d9c8a86a54deb039862ab3ae59eba6f22c7374fa3d27f WatchSource:0}: Error finding container ab6d8be7e1afb6e3ae6d9c8a86a54deb039862ab3ae59eba6f22c7374fa3d27f: Status 404 returned error can't find the container with id ab6d8be7e1afb6e3ae6d9c8a86a54deb039862ab3ae59eba6f22c7374fa3d27f
Sep 08 18:44:12.467544 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:12.467522    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:44:13.298525 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:13.298490    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:13.305917 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:13.305885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j" event={"ID":"55216b79-6258-4fc1-8568-4d7f6559d00c","Type":"ContainerStarted","Data":"ab6d8be7e1afb6e3ae6d9c8a86a54deb039862ab3ae59eba6f22c7374fa3d27f"}
Sep 08 18:44:14.642318 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:14.642283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:14.691385 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:14.691332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:14.819446 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:14.819416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:14.923313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:14.923226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:15.392224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:15.392184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:15.445632 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:15.445601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:19.705920 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:19.705890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:19.803888 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:19.803862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:20.604992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.604958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:20.746495 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.746461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:20.852320 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.852286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:20.891919 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.891853    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"]
Sep 08 18:44:20.896213 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.896194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:20.898096 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.898073    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 18:44:20.898306 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.898287    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 18:44:20.898435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.898291    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 18:44:20.898435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.898362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z6qp9\""
Sep 08 18:44:20.898435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.898413    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 18:44:20.907667 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:20.907642    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"]
Sep 08 18:44:21.049104 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.049070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6948f32a-4e7b-4ee9-bed1-398916c7995f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.049260 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.049117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6948f32a-4e7b-4ee9-bed1-398916c7995f-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.049260 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.049135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6948f32a-4e7b-4ee9-bed1-398916c7995f-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.049260 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.049210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ccvj\" (UniqueName: \"kubernetes.io/projected/6948f32a-4e7b-4ee9-bed1-398916c7995f-kube-api-access-4ccvj\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.049404 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.049268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6948f32a-4e7b-4ee9-bed1-398916c7995f-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.075693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.075666    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:21.149970 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.149886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6948f32a-4e7b-4ee9-bed1-398916c7995f-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.149970 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.149923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6948f32a-4e7b-4ee9-bed1-398916c7995f-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.149970 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.149950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ccvj\" (UniqueName: \"kubernetes.io/projected/6948f32a-4e7b-4ee9-bed1-398916c7995f-kube-api-access-4ccvj\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.150267 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.150076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6948f32a-4e7b-4ee9-bed1-398916c7995f-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.150267 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.150131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6948f32a-4e7b-4ee9-bed1-398916c7995f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.150416 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.150298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6948f32a-4e7b-4ee9-bed1-398916c7995f-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.152428 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.152406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6948f32a-4e7b-4ee9-bed1-398916c7995f-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.152636 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.152617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6948f32a-4e7b-4ee9-bed1-398916c7995f-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.152682 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.152637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6948f32a-4e7b-4ee9-bed1-398916c7995f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.157493 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.157468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ccvj\" (UniqueName: \"kubernetes.io/projected/6948f32a-4e7b-4ee9-bed1-398916c7995f-kube-api-access-4ccvj\") pod \"opendatahub-operator-controller-manager-67894858cf-9hdrt\" (UID: \"6948f32a-4e7b-4ee9-bed1-398916c7995f\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.175233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.175208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:21.206186 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.206154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:21.276131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.276102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:21.375332 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.375303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:21.466398 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.466303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:21.482785 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.482595    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"]
Sep 08 18:44:21.486200 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:44:21.486176    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6948f32a_4e7b_4ee9_bed1_398916c7995f.slice/crio-a67e38a8d2070f62d77875b954cbb8e727a7d3b106b49313f4f9b7c686b3dea9 WatchSource:0}: Error finding container a67e38a8d2070f62d77875b954cbb8e727a7d3b106b49313f4f9b7c686b3dea9: Status 404 returned error can't find the container with id a67e38a8d2070f62d77875b954cbb8e727a7d3b106b49313f4f9b7c686b3dea9
Sep 08 18:44:21.533915 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.533880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:21.623353 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:21.623310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:22.060436 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:22.060326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:22.177864 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:22.177821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:22.370884 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:22.370841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt" event={"ID":"6948f32a-4e7b-4ee9-bed1-398916c7995f","Type":"ContainerStarted","Data":"a67e38a8d2070f62d77875b954cbb8e727a7d3b106b49313f4f9b7c686b3dea9"}
Sep 08 18:44:24.011873 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:24.011836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:24.110814 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:24.110761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:25.268247 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:25.268214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:25.369061 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:25.369025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:25.391555 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:25.391523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt" event={"ID":"6948f32a-4e7b-4ee9-bed1-398916c7995f","Type":"ContainerStarted","Data":"cc674e617fc56d56ed6cc113f78b9a44da2d19c6112073a4770ddab7e9dc5470"}
Sep 08 18:44:25.485841 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:25.485809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:26.397212 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:26.397182    2579 generic.go:356] "Generic (PLEG): container finished" podID="6948f32a-4e7b-4ee9-bed1-398916c7995f" containerID="cc674e617fc56d56ed6cc113f78b9a44da2d19c6112073a4770ddab7e9dc5470" exitCode=0
Sep 08 18:44:26.397607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:26.397260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt" event={"ID":"6948f32a-4e7b-4ee9-bed1-398916c7995f","Type":"ContainerDied","Data":"cc674e617fc56d56ed6cc113f78b9a44da2d19c6112073a4770ddab7e9dc5470"}
Sep 08 18:44:27.403770 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:27.403735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt" event={"ID":"6948f32a-4e7b-4ee9-bed1-398916c7995f","Type":"ContainerStarted","Data":"5a9ff9f6045d454bc0b7c832d0e9a8c4f214b71dd92f7db15cf275ae847d15c0"}
Sep 08 18:44:27.404243 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:27.403848    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:27.405209 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:27.405186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j" event={"ID":"55216b79-6258-4fc1-8568-4d7f6559d00c","Type":"ContainerStarted","Data":"43aa64a545060aae35ccb8c9920d6e6f83e5fa6f395fe1d0e436f62d63e8c72a"}
Sep 08 18:44:27.405331 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:27.405298    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:27.424789 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:27.424746    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt" podStartSLOduration=2.537940496 podStartE2EDuration="7.424732636s" totalImagesPullingTime="3.594876751s" totalInitContainerRuntime="1.291915389s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:27.422723015 +0000 UTC m=+627.776816347" watchObservedRunningTime="2026-09-08 18:44:27.424732636 +0000 UTC m=+627.778825969"
Sep 08 18:44:27.440437 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:27.440396    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j" podStartSLOduration=2.515883229 podStartE2EDuration="16.440382889s" totalImagesPullingTime="13.92449966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:27.439247799 +0000 UTC m=+627.793341131" watchObservedRunningTime="2026-09-08 18:44:27.440382889 +0000 UTC m=+627.794476219"
Sep 08 18:44:32.907759 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:32.907727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:33.009358 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:33.009312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:35.741271 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:35.741235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:35.841793 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:35.841763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:38.411163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:38.411131    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-f5m5j"
Sep 08 18:44:38.411591 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:38.411297    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-9hdrt"
Sep 08 18:44:40.725548 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:40.725522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:40.840146 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:40.840116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:41.072720 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.072639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:41.146392 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.146355    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"]
Sep 08 18:44:41.157398 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.157368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.160589 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.160561    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 18:44:41.160589 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.160561    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gtwxl\""
Sep 08 18:44:41.160914 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.160898    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 18:44:41.169890 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.169868    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"]
Sep 08 18:44:41.228331 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.228300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tls-certs\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.228501 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.228375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tmp\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.228501 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.228483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwml\" (UniqueName: \"kubernetes.io/projected/9a4ebde5-218d-466c-91fe-2e68cc635bf7-kube-api-access-chwml\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.329780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.329746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwml\" (UniqueName: \"kubernetes.io/projected/9a4ebde5-218d-466c-91fe-2e68cc635bf7-kube-api-access-chwml\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.329955 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.329823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tls-certs\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.329955 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.329868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tmp\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.330076 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:44:41.329957    2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 08 18:44:41.330076 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:44:41.330054    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tls-certs podName:9a4ebde5-218d-466c-91fe-2e68cc635bf7 nodeName:}" failed. No retries permitted until 2026-09-08 18:44:41.830038163 +0000 UTC m=+642.184131474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tls-certs") pod "kube-auth-proxy-54b8878898-fz9t7" (UID: "9a4ebde5-218d-466c-91fe-2e68cc635bf7") : secret "kube-auth-proxy-tls" not found
Sep 08 18:44:41.332093 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.332070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tmp\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.340225 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.340203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwml\" (UniqueName: \"kubernetes.io/projected/9a4ebde5-218d-466c-91fe-2e68cc635bf7-kube-api-access-chwml\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.491890 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.491854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:41.833443 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.833418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tls-certs\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:41.835833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:41.835808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4ebde5-218d-466c-91fe-2e68cc635bf7-tls-certs\") pod \"kube-auth-proxy-54b8878898-fz9t7\" (UID: \"9a4ebde5-218d-466c-91fe-2e68cc635bf7\") " pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:42.067106 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:42.067053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"
Sep 08 18:44:42.339797 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:42.339760    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54b8878898-fz9t7"]
Sep 08 18:44:42.343398 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:44:42.343363    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4ebde5_218d_466c_91fe_2e68cc635bf7.slice/crio-649c461176f4008f1d2d73d1686f39ba13a825c0c58dd387a58c75e12940c8fd WatchSource:0}: Error finding container 649c461176f4008f1d2d73d1686f39ba13a825c0c58dd387a58c75e12940c8fd: Status 404 returned error can't find the container with id 649c461176f4008f1d2d73d1686f39ba13a825c0c58dd387a58c75e12940c8fd
Sep 08 18:44:42.384279 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:42.384211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:42.442788 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:42.442756    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:42.467374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:42.467328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7" event={"ID":"9a4ebde5-218d-466c-91fe-2e68cc635bf7","Type":"ContainerStarted","Data":"649c461176f4008f1d2d73d1686f39ba13a825c0c58dd387a58c75e12940c8fd"}
Sep 08 18:44:43.220411 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:43.220328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:45.483729 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:45.483687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7" event={"ID":"9a4ebde5-218d-466c-91fe-2e68cc635bf7","Type":"ContainerStarted","Data":"4f4c9a4486f22b311deb97066ca5067308c464fac67d3c742c447085b52ec48e"}
Sep 08 18:44:45.497106 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:45.497051    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54b8878898-fz9t7" podStartSLOduration=1.457008774 podStartE2EDuration="4.49703074s" totalImagesPullingTime="3.040021966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:45.496315685 +0000 UTC m=+645.850409017" watchObservedRunningTime="2026-09-08 18:44:45.49703074 +0000 UTC m=+645.851124091"
Sep 08 18:44:51.814362 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:51.814312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:51.873795 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:51.873752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:51.976020 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:51.975986    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:52.338525 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:52.338488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:52.437703 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:52.437670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:52.535742 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:52.535709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:52.693115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:52.693034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:52.745933 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:52.745890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:52.990691 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:52.990618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:53.334549 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:53.334516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:44:53.439238 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:44:53.439205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:02.280943 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:02.280901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:02.419753 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:02.419713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:02.578980 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:02.578884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:02.678192 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:02.678159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:03.678768 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:03.678733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:03.764588 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:03.764546    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:03.831574 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:03.831545    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:31.039640 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:31.039605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:31.087694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:31.087656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:51.467501 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:51.467468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:51.517545 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:51.517506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:53.037441 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:53.037402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:45:53.131759 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:45:53.131730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:05.361929 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:05.361890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:06.150823 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:06.150793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:09.254505 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:09.254469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:09.361318 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:09.361288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:09.458307 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:09.458277    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:11.394233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:11.394197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:11.562717 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:11.562687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:13.549300 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:13.549266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:13.796075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:13.796030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:13.962646 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:13.962614    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:14.549512 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:14.549477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:15.204200 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:15.204163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:15.308625 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:15.308596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:19.155375 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:19.155323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:19.263146 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:19.263113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:19.765445 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:19.765408    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:19.831817 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:19.831792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:23.154634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:23.154602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:23.254762 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:23.254730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:24.749021 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.748983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:24.963185 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.963153    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"]
Sep 08 18:46:24.966667 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.966644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:24.968396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.968375    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 18:46:24.968979 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.968963    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 18:46:24.969060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.968977    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 18:46:24.969060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.968986    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 18:46:24.969060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.969011    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mqx9b\""
Sep 08 18:46:24.973677 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:24.973396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"]
Sep 08 18:46:25.129687 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.129658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f662d319-5f4b-4fae-80ed-02eae3098835-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.129864 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.129700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f662d319-5f4b-4fae-80ed-02eae3098835-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.129864 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.129727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjwd\" (UniqueName: \"kubernetes.io/projected/f662d319-5f4b-4fae-80ed-02eae3098835-kube-api-access-6xjwd\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.183363 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.183319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:25.230525 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.230490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f662d319-5f4b-4fae-80ed-02eae3098835-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.230699 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.230539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f662d319-5f4b-4fae-80ed-02eae3098835-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.230699 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.230634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjwd\" (UniqueName: \"kubernetes.io/projected/f662d319-5f4b-4fae-80ed-02eae3098835-kube-api-access-6xjwd\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.231113 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.231087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f662d319-5f4b-4fae-80ed-02eae3098835-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.232935 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.232916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f662d319-5f4b-4fae-80ed-02eae3098835-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.237644 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.237620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjwd\" (UniqueName: \"kubernetes.io/projected/f662d319-5f4b-4fae-80ed-02eae3098835-kube-api-access-6xjwd\") pod \"kuadrant-console-plugin-fbdd4f445-8njjp\" (UID: \"f662d319-5f4b-4fae-80ed-02eae3098835\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.284168 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.284143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"
Sep 08 18:46:25.284753 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.284719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:25.554509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.554410    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp"]
Sep 08 18:46:25.557206 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:46:25.557170    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf662d319_5f4b_4fae_80ed_02eae3098835.slice/crio-159b82562ae280796a8d89e57ffc4cc796aa61d66c134572a14327187ed48d7d WatchSource:0}: Error finding container 159b82562ae280796a8d89e57ffc4cc796aa61d66c134572a14327187ed48d7d: Status 404 returned error can't find the container with id 159b82562ae280796a8d89e57ffc4cc796aa61d66c134572a14327187ed48d7d
Sep 08 18:46:25.910639 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:25.910603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp" event={"ID":"f662d319-5f4b-4fae-80ed-02eae3098835","Type":"ContainerStarted","Data":"159b82562ae280796a8d89e57ffc4cc796aa61d66c134572a14327187ed48d7d"}
Sep 08 18:46:30.937651 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:30.937615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp" event={"ID":"f662d319-5f4b-4fae-80ed-02eae3098835","Type":"ContainerStarted","Data":"02aecd78eee4578383138bd33dfe694a63982424cee3d4abab100fc1d29d9165"}
Sep 08 18:46:30.953889 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:30.953839    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8njjp" podStartSLOduration=2.069899515 podStartE2EDuration="6.95382337s" totalImagesPullingTime="4.883923855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:30.951985625 +0000 UTC m=+751.306078961" watchObservedRunningTime="2026-09-08 18:46:30.95382337 +0000 UTC m=+751.307916702"
Sep 08 18:46:31.094904 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:31.094873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:31.191834 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:31.191744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:31.292212 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:31.292182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:39.039867 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:39.039824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:39.139418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:39.139382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:39.404267 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:39.404234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:39.505630 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:39.505599    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:42.816843 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:42.816805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:42.919511 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:42.919477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:44.401960 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:44.401926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:44.501935 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:44.501901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:50.298414 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:50.298378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:50.393013 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:50.392976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:50.788999 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:50.788960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:50.888391 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:50.888358    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:46:51.002612 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:46:51.002580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:04.365292 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:04.365260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:04.697948 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:04.697860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:19.860293 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:19.860257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:19.976144 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:19.976114    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:20.076314 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.076280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:20.257063 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.256977    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:47:20.277248 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.277218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:47:20.277425 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.277305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:20.279651 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.279625    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 18:47:20.280718 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.280693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:47:20.408576 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.408536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a37071b-4855-4b9a-be96-ec1899815bbc-config-file\") pod \"limitador-limitador-84994f7ffc-qrpc7\" (UID: \"1a37071b-4855-4b9a-be96-ec1899815bbc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:20.408763 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.408678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfk2d\" (UniqueName: \"kubernetes.io/projected/1a37071b-4855-4b9a-be96-ec1899815bbc-kube-api-access-mfk2d\") pod \"limitador-limitador-84994f7ffc-qrpc7\" (UID: \"1a37071b-4855-4b9a-be96-ec1899815bbc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:20.509908 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.509829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfk2d\" (UniqueName: \"kubernetes.io/projected/1a37071b-4855-4b9a-be96-ec1899815bbc-kube-api-access-mfk2d\") pod \"limitador-limitador-84994f7ffc-qrpc7\" (UID: \"1a37071b-4855-4b9a-be96-ec1899815bbc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:20.509908 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.509871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a37071b-4855-4b9a-be96-ec1899815bbc-config-file\") pod \"limitador-limitador-84994f7ffc-qrpc7\" (UID: \"1a37071b-4855-4b9a-be96-ec1899815bbc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:20.510579 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.510560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a37071b-4855-4b9a-be96-ec1899815bbc-config-file\") pod \"limitador-limitador-84994f7ffc-qrpc7\" (UID: \"1a37071b-4855-4b9a-be96-ec1899815bbc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:20.517833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.517809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfk2d\" (UniqueName: \"kubernetes.io/projected/1a37071b-4855-4b9a-be96-ec1899815bbc-kube-api-access-mfk2d\") pod \"limitador-limitador-84994f7ffc-qrpc7\" (UID: \"1a37071b-4855-4b9a-be96-ec1899815bbc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:20.587905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.587867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:20.693011 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.692981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:20.769248 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.769170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:20.857104 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:20.857078    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:47:20.859802 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:47:20.859765    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a37071b_4855_4b9a_be96_ec1899815bbc.slice/crio-4e0c0101a3d870e080b4a1bf2f522e56cad893b6628491c052be3b79291b8b04 WatchSource:0}: Error finding container 4e0c0101a3d870e080b4a1bf2f522e56cad893b6628491c052be3b79291b8b04: Status 404 returned error can't find the container with id 4e0c0101a3d870e080b4a1bf2f522e56cad893b6628491c052be3b79291b8b04
Sep 08 18:47:21.157841 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:21.157803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7" event={"ID":"1a37071b-4855-4b9a-be96-ec1899815bbc","Type":"ContainerStarted","Data":"4e0c0101a3d870e080b4a1bf2f522e56cad893b6628491c052be3b79291b8b04"}
Sep 08 18:47:24.178832 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:24.178796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7" event={"ID":"1a37071b-4855-4b9a-be96-ec1899815bbc","Type":"ContainerStarted","Data":"2224d59a9c9b9117467b45fae86e3472b730afec80dd006a5836c884f9916463"}
Sep 08 18:47:24.179318 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:24.178873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:24.193949 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:24.193901    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7" podStartSLOduration=1.05180603 podStartE2EDuration="4.193884565s" totalImagesPullingTime="3.142078535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:24.193278004 +0000 UTC m=+804.547371338" watchObservedRunningTime="2026-09-08 18:47:24.193884565 +0000 UTC m=+804.547977898"
Sep 08 18:47:35.184183 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:35.184142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"
Sep 08 18:47:36.429333 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:36.429296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:36.522847 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:36.522813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:36.947304 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:36.947267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:49.421780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:49.421749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:49.635536 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:49.635503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:56.050584 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:56.050540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:56.126989 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:56.126959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:56.181362 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:56.181298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:56.265711 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:56.265683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:56.319601 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:56.319513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:59.399363 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:59.399314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:47:59.497728 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:47:59.497696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:30.719956 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:30.719920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:30.820619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:30.820581    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:30.974382 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:30.974262    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:32.110745 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:32.110715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:32.210909 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:32.210878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:32.333451 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:32.333416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:32.784275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:32.784246    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:32.876967 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:32.876936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:32.947496 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:32.947454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:32.997068 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:32.997035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:48:43.398444 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:43.398404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:43.504854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:43.504813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:55.076051 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.076016    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"]
Sep 08 18:48:55.079707 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.079685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.082589 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.082563    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 18:48:55.082745 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.082586    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-htg2r\""
Sep 08 18:48:55.082745 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.082617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 18:48:55.082745 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.082713    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 18:48:55.092881 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.092852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"]
Sep 08 18:48:55.146154 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.146125    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:48:55.241054 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.241023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxvnx\" (UniqueName: \"kubernetes.io/projected/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-kube-api-access-rxvnx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.241212 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.241066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.241212 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.241084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.241212 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.241110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.241387 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.241262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.241387 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.241302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342152 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342152 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342419 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342419 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342419 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342419 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxvnx\" (UniqueName: \"kubernetes.io/projected/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-kube-api-access-rxvnx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342641 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342697 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.342750 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.342733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.344492 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.344472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.344782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.344746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.349387 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.349367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxvnx\" (UniqueName: \"kubernetes.io/projected/dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc-kube-api-access-rxvnx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j\" (UID: \"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.393301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.393269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:48:55.663044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.663016    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"]
Sep 08 18:48:55.665590 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:48:55.665563    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd90c2ea_59cb_4c8f_b921_ec593ffd0ffc.slice/crio-d0755c2a62ea8843b9bbe4ae2d53639939a8189b99d9a2094c6834b1b60a405e WatchSource:0}: Error finding container d0755c2a62ea8843b9bbe4ae2d53639939a8189b99d9a2094c6834b1b60a405e: Status 404 returned error can't find the container with id d0755c2a62ea8843b9bbe4ae2d53639939a8189b99d9a2094c6834b1b60a405e
Sep 08 18:48:55.682970 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:55.682948    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:48:56.573612 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:56.573564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j" event={"ID":"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc","Type":"ContainerStarted","Data":"d0755c2a62ea8843b9bbe4ae2d53639939a8189b99d9a2094c6834b1b60a405e"}
Sep 08 18:48:57.579851 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:57.579812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j" event={"ID":"dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc","Type":"ContainerStarted","Data":"1264143966eefed6036bb806a69420154d88c34d396d2e4f9de5229669f018c2"}
Sep 08 18:48:57.595316 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:48:57.595225    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j" podStartSLOduration=1.43178044 podStartE2EDuration="2.595206961s" totalImagesPullingTime="1.163426521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:48:57.594532525 +0000 UTC m=+897.948625858" watchObservedRunningTime="2026-09-08 18:48:57.595206961 +0000 UTC m=+897.949300298"
Sep 08 18:49:00.105423 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:00.105393    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:49:00.105423 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:00.105425    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:49:05.394460 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:05.394421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:49:05.396779 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:05.394495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:49:05.407021 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:05.406994    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:49:05.626636 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:05.626602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j"
Sep 08 18:49:23.388615 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:23.388577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:49:24.070902 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:24.070871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:49:28.311419 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:28.311386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:49:29.474468 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:29.474431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:49:33.594336 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:33.594300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:49:34.977845 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:34.977808    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:49:39.484895 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.484863    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"]
Sep 08 18:49:39.487960 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.487940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.490130 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.490109    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 08 18:49:39.502810 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.502782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"]
Sep 08 18:49:39.523331 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.523307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:49:39.610760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.610718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.610869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.610778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.610869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.610829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.610869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.610856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.610974 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.610873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be70e0d4-c0ef-415b-b234-a2f24546b4ac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.610974 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.610890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x96v\" (UniqueName: \"kubernetes.io/projected/be70e0d4-c0ef-415b-b234-a2f24546b4ac-kube-api-access-4x96v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712312 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712312 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712524 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712524 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be70e0d4-c0ef-415b-b234-a2f24546b4ac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712524 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x96v\" (UniqueName: \"kubernetes.io/projected/be70e0d4-c0ef-415b-b234-a2f24546b4ac-kube-api-access-4x96v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712670 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712788 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712788 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.712962 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.712805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.714694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.714675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be70e0d4-c0ef-415b-b234-a2f24546b4ac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.714981 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.714960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be70e0d4-c0ef-415b-b234-a2f24546b4ac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.719876 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.719851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x96v\" (UniqueName: \"kubernetes.io/projected/be70e0d4-c0ef-415b-b234-a2f24546b4ac-kube-api-access-4x96v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28\" (UID: \"be70e0d4-c0ef-415b-b234-a2f24546b4ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:39.803765 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:39.803696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:40.073184 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:40.073162    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"]
Sep 08 18:49:40.075247 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:49:40.075220    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe70e0d4_c0ef_415b_b234_a2f24546b4ac.slice/crio-bf9830fc47a9cf0476bf16885f9d64f4dd6a51e2ee5354f74ecd8af34ab39abc WatchSource:0}: Error finding container bf9830fc47a9cf0476bf16885f9d64f4dd6a51e2ee5354f74ecd8af34ab39abc: Status 404 returned error can't find the container with id bf9830fc47a9cf0476bf16885f9d64f4dd6a51e2ee5354f74ecd8af34ab39abc
Sep 08 18:49:40.076540 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:40.076520    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:49:40.766475 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:40.766441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28" event={"ID":"be70e0d4-c0ef-415b-b234-a2f24546b4ac","Type":"ContainerStarted","Data":"7878f24415a600c748be6a3af01ef1f3d4c7085336e73795a3561532b9da2242"}
Sep 08 18:49:40.766846 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:40.766494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28" event={"ID":"be70e0d4-c0ef-415b-b234-a2f24546b4ac","Type":"ContainerStarted","Data":"bf9830fc47a9cf0476bf16885f9d64f4dd6a51e2ee5354f74ecd8af34ab39abc"}
Sep 08 18:49:40.784706 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:40.784656    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28" podStartSLOduration=1.6057046879999999 podStartE2EDuration="1.784641136s" totalImagesPullingTime="178.936448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:40.782415783 +0000 UTC m=+941.136509120" watchObservedRunningTime="2026-09-08 18:49:40.784641136 +0000 UTC m=+941.138734469"
Sep 08 18:49:40.978972 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:40.978932    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:49:49.804452 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:49.804420    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:49.804893 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:49.804699    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:49.816715 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:49.816692    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:50.825944 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:50.825915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28"
Sep 08 18:49:53.117633 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:53.117597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:49:54.876357 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:49:54.876308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:50:01.374328 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.374285    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"]
Sep 08 18:50:01.377853 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.377833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.379881 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.379856    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 18:50:01.387621 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.387596    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"]
Sep 08 18:50:01.405371 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.405331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.405510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.405402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76775\" (UniqueName: \"kubernetes.io/projected/619602d0-e639-41c9-8a5e-bc2cca813481-kube-api-access-76775\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.405510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.405485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/619602d0-e639-41c9-8a5e-bc2cca813481-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.405510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.405503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.405624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.405526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.405624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.405543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.413235 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.413209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:50:01.506884 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.506842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.507081 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.506889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76775\" (UniqueName: \"kubernetes.io/projected/619602d0-e639-41c9-8a5e-bc2cca813481-kube-api-access-76775\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.507081 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.506938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/619602d0-e639-41c9-8a5e-bc2cca813481-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.507081 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.506958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.507081 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.506983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.507081 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.507005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.507446 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.507426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.507527 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.507497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.507527 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.507509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.509195 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.509171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/619602d0-e639-41c9-8a5e-bc2cca813481-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.509998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.509977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/619602d0-e639-41c9-8a5e-bc2cca813481-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.513825 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.513795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76775\" (UniqueName: \"kubernetes.io/projected/619602d0-e639-41c9-8a5e-bc2cca813481-kube-api-access-76775\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-96p4j\" (UID: \"619602d0-e639-41c9-8a5e-bc2cca813481\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.688440 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.688332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:01.957510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:01.957295    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"]
Sep 08 18:50:01.959793 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:50:01.959768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod619602d0_e639_41c9_8a5e_bc2cca813481.slice/crio-422b61398b61f3f55a09a8d441499817dd3a68463ae3924696e6d9e1390fa59d WatchSource:0}: Error finding container 422b61398b61f3f55a09a8d441499817dd3a68463ae3924696e6d9e1390fa59d: Status 404 returned error can't find the container with id 422b61398b61f3f55a09a8d441499817dd3a68463ae3924696e6d9e1390fa59d
Sep 08 18:50:02.869312 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:02.869277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j" event={"ID":"619602d0-e639-41c9-8a5e-bc2cca813481","Type":"ContainerStarted","Data":"7d08ea098d5b7c5681f87029cf5be227ed84cc9a420a9d929494dbd0ad232bdb"}
Sep 08 18:50:02.869785 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:02.869321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j" event={"ID":"619602d0-e639-41c9-8a5e-bc2cca813481","Type":"ContainerStarted","Data":"422b61398b61f3f55a09a8d441499817dd3a68463ae3924696e6d9e1390fa59d"}
Sep 08 18:50:02.885532 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:02.885484    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j" podStartSLOduration=1.680505926 podStartE2EDuration="1.8854683s" totalImagesPullingTime="204.962374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:50:02.883376811 +0000 UTC m=+963.237470140" watchObservedRunningTime="2026-09-08 18:50:02.8854683 +0000 UTC m=+963.239561633"
Sep 08 18:50:03.775650 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:03.775611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:50:11.689413 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:11.689375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:11.689413 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:11.689421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:11.701704 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:11.701673    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:50:11.758525 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:11.758492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:50:11.813027 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:11.812998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:50:11.925874 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:50:11.925842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-96p4j"
Sep 08 18:51:27.387743 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:27.387708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:27.491802 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:27.491766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:29.631647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:29.631612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:29.730823 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:29.730787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:48.293734 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:48.293699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:48.395557 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:48.395522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:48.637919 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:48.637884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:49.188686 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:49.188652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:49.929724 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:49.929693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:50.099567 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:50.099539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:50.449694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:50.449648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:51.513104 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:51.513074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:51.632107 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:51.632076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:52.127982 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:52.127953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:51:52.900664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:52.900625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:53.055317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:53.055267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:53.530526 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:53.530489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:53.850083 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:53.850050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:53.987725 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:53.987683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:55.432531 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:55.432493    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-d3a248"]
Sep 08 18:51:55.436139 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:55.436116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-d3a248"
Sep 08 18:51:55.439506 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:55.439487    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgjvd\""
Sep 08 18:51:55.447169 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:55.447147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-d3a248"]
Sep 08 18:51:55.506116 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:55.506078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zptvx\" (UniqueName: \"kubernetes.io/projected/154fe63a-0f77-4eb2-b1bb-719413b622ef-kube-api-access-zptvx\") pod \"test-curl-7704-d3a248\" (UID: \"154fe63a-0f77-4eb2-b1bb-719413b622ef\") " pod="opendatahub/test-curl-7704-d3a248"
Sep 08 18:51:55.607197 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:55.607162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zptvx\" (UniqueName: \"kubernetes.io/projected/154fe63a-0f77-4eb2-b1bb-719413b622ef-kube-api-access-zptvx\") pod \"test-curl-7704-d3a248\" (UID: \"154fe63a-0f77-4eb2-b1bb-719413b622ef\") " pod="opendatahub/test-curl-7704-d3a248"
Sep 08 18:51:55.620411 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:55.620377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zptvx\" (UniqueName: \"kubernetes.io/projected/154fe63a-0f77-4eb2-b1bb-719413b622ef-kube-api-access-zptvx\") pod \"test-curl-7704-d3a248\" (UID: \"154fe63a-0f77-4eb2-b1bb-719413b622ef\") " pod="opendatahub/test-curl-7704-d3a248"
Sep 08 18:51:55.745065 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:55.744990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-d3a248"
Sep 08 18:51:56.026127 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:56.026103    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-d3a248"]
Sep 08 18:51:56.028495 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:51:56.028470    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod154fe63a_0f77_4eb2_b1bb_719413b622ef.slice/crio-62c92df539b763642ec562077fe4b862ee6a7622d68d9b3e9cca5fcdc26de7c1 WatchSource:0}: Error finding container 62c92df539b763642ec562077fe4b862ee6a7622d68d9b3e9cca5fcdc26de7c1: Status 404 returned error can't find the container with id 62c92df539b763642ec562077fe4b862ee6a7622d68d9b3e9cca5fcdc26de7c1
Sep 08 18:51:56.382749 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:56.382714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-d3a248" event={"ID":"154fe63a-0f77-4eb2-b1bb-719413b622ef","Type":"ContainerStarted","Data":"62c92df539b763642ec562077fe4b862ee6a7622d68d9b3e9cca5fcdc26de7c1"}
Sep 08 18:51:57.407714 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:57.407674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:57.562699 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:57.562670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:57.786726 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:57.786648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:58.171275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:58.171244    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:58.395100 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:58.395068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-d3a248" event={"ID":"154fe63a-0f77-4eb2-b1bb-719413b622ef","Type":"ContainerStarted","Data":"a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30"}
Sep 08 18:51:59.158278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:59.158177    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7704-d3a248" podStartSLOduration=1.932064044 podStartE2EDuration="4.158133235s" totalImagesPullingTime="2.226069191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:58.429480672 +0000 UTC m=+1078.783574006" watchObservedRunningTime="2026-09-08 18:51:59.158133235 +0000 UTC m=+1079.512226558"
Sep 08 18:51:59.159646 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:59.159626    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-d3a248"]
Sep 08 18:51:59.168749 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:59.168719    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-d3a248"]
Sep 08 18:51:59.191132 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:59.191105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:51:59.399928 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:51:59.399896    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7704-d3a248" container="test-curl-7704-d3a248"
Sep 08 18:51:59.715051 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:51:59.715017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:00.036492 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:00.036406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:00.254101 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:00.254066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:00.404049 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:00.404012    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7704-d3a248" podUID="154fe63a-0f77-4eb2-b1bb-719413b622ef" containerName="test-curl-7704-d3a248" containerID="cri-o://a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30" gracePeriod=2
Sep 08 18:52:00.597469 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:00.597440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:00.692804 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:00.692725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:01.713786 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.713750    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"]
Sep 08 18:52:01.714397 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.714376    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154fe63a-0f77-4eb2-b1bb-719413b622ef" containerName="test-curl-7704-d3a248"
Sep 08 18:52:01.714453 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.714400    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="154fe63a-0f77-4eb2-b1bb-719413b622ef" containerName="test-curl-7704-d3a248"
Sep 08 18:52:01.714518 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.714505    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="154fe63a-0f77-4eb2-b1bb-719413b622ef" containerName="test-curl-7704-d3a248"
Sep 08 18:52:01.717875 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.717852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.721474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.721451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ffda9270-serving-cert\""
Sep 08 18:52:01.721603 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.721476    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:52:01.721603 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.721480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 18:52:01.721814 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.721796    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:52:01.721814 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.721809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:52:01.730081 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.730049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"]
Sep 08 18:52:01.774679 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.774643    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:01.867630 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.867591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98kx\" (UniqueName: \"kubernetes.io/projected/405a395a-5026-45e4-bd48-f921f6ef7885-kube-api-access-t98kx\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.867818 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.867649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-metrics-certs\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.867818 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.867691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-maas-api-tls\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.867818 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.867706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a395a-5026-45e4-bd48-f921f6ef7885-openshift-service-ca\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.875992 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.875964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:01.969369 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.969246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-maas-api-tls\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.969369 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.969286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a395a-5026-45e4-bd48-f921f6ef7885-openshift-service-ca\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.969584 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.969411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98kx\" (UniqueName: \"kubernetes.io/projected/405a395a-5026-45e4-bd48-f921f6ef7885-kube-api-access-t98kx\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.969584 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.969471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-metrics-certs\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.970076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.970049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a395a-5026-45e4-bd48-f921f6ef7885-openshift-service-ca\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.971916 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.971889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-maas-api-tls\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.972033 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.971964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-metrics-certs\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:01.973005 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.972981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:01.978381 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:01.978355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98kx\" (UniqueName: \"kubernetes.io/projected/405a395a-5026-45e4-bd48-f921f6ef7885-kube-api-access-t98kx\") pod \"maas-api-e2e-ait-ffda9270-686974c47f-fmc8d\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:02.028462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.028418    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:02.310076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.309991    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"]
Sep 08 18:52:02.313373 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:52:02.313325    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405a395a_5026_45e4_bd48_f921f6ef7885.slice/crio-b73b5f64542c4cd8f7232a9971f6b267067477370f2bf34444316e713c070b07 WatchSource:0}: Error finding container b73b5f64542c4cd8f7232a9971f6b267067477370f2bf34444316e713c070b07: Status 404 returned error can't find the container with id b73b5f64542c4cd8f7232a9971f6b267067477370f2bf34444316e713c070b07
Sep 08 18:52:02.398744 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.398704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:02.416650 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.416604    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d" event={"ID":"405a395a-5026-45e4-bd48-f921f6ef7885","Type":"ContainerStarted","Data":"b73b5f64542c4cd8f7232a9971f6b267067477370f2bf34444316e713c070b07"}
Sep 08 18:52:02.548818 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.548777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:02.563143 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.563071    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"]
Sep 08 18:52:02.566665 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.566641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:02.569719 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.569695    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ffda9270-dockercfg-vjwfd\""
Sep 08 18:52:02.569871 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.569850    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ffda9270\""
Sep 08 18:52:02.585728 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.585703    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-d3a248"
Sep 08 18:52:02.588043 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.588014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"]
Sep 08 18:52:02.675230 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.675201    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zptvx\" (UniqueName: \"kubernetes.io/projected/154fe63a-0f77-4eb2-b1bb-719413b622ef-kube-api-access-zptvx\") pod \"154fe63a-0f77-4eb2-b1bb-719413b622ef\" (UID: \"154fe63a-0f77-4eb2-b1bb-719413b622ef\") "
Sep 08 18:52:02.675434 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.675415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09566ff2-c460-4ac1-b944-b6547debe960-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n\" (UID: \"09566ff2-c460-4ac1-b944-b6547debe960\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:02.675500 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.675484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9bq\" (UniqueName: \"kubernetes.io/projected/09566ff2-c460-4ac1-b944-b6547debe960-kube-api-access-8g9bq\") pod \"payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n\" (UID: \"09566ff2-c460-4ac1-b944-b6547debe960\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:02.677542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.677511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154fe63a-0f77-4eb2-b1bb-719413b622ef-kube-api-access-zptvx" (OuterVolumeSpecName: "kube-api-access-zptvx") pod "154fe63a-0f77-4eb2-b1bb-719413b622ef" (UID: "154fe63a-0f77-4eb2-b1bb-719413b622ef"). InnerVolumeSpecName "kube-api-access-zptvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:52:02.776665 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.776623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09566ff2-c460-4ac1-b944-b6547debe960-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n\" (UID: \"09566ff2-c460-4ac1-b944-b6547debe960\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:02.777036 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.776687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9bq\" (UniqueName: \"kubernetes.io/projected/09566ff2-c460-4ac1-b944-b6547debe960-kube-api-access-8g9bq\") pod \"payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n\" (UID: \"09566ff2-c460-4ac1-b944-b6547debe960\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:02.777036 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.776815    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zptvx\" (UniqueName: \"kubernetes.io/projected/154fe63a-0f77-4eb2-b1bb-719413b622ef-kube-api-access-zptvx\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:52:02.777328 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.777307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09566ff2-c460-4ac1-b944-b6547debe960-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n\" (UID: \"09566ff2-c460-4ac1-b944-b6547debe960\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:02.788550 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.788528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9bq\" (UniqueName: \"kubernetes.io/projected/09566ff2-c460-4ac1-b944-b6547debe960-kube-api-access-8g9bq\") pod \"payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n\" (UID: \"09566ff2-c460-4ac1-b944-b6547debe960\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:02.882148 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:02.882112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:03.161060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.161032    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"]
Sep 08 18:52:03.163111 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:52:03.163085    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09566ff2_c460_4ac1_b944_b6547debe960.slice/crio-a7cd4f457406bd62af5770446358fa5d963e3b1870ad81723a35f466e6966c14 WatchSource:0}: Error finding container a7cd4f457406bd62af5770446358fa5d963e3b1870ad81723a35f466e6966c14: Status 404 returned error can't find the container with id a7cd4f457406bd62af5770446358fa5d963e3b1870ad81723a35f466e6966c14
Sep 08 18:52:03.423439 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.423355    2579 generic.go:356] "Generic (PLEG): container finished" podID="154fe63a-0f77-4eb2-b1bb-719413b622ef" containerID="a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30" exitCode=137
Sep 08 18:52:03.423439 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.423398    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-d3a248"
Sep 08 18:52:03.423668 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.423458    2579 scope.go:117] "RemoveContainer" containerID="a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30"
Sep 08 18:52:03.424948 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.424923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n" event={"ID":"09566ff2-c460-4ac1-b944-b6547debe960","Type":"ContainerStarted","Data":"a7cd4f457406bd62af5770446358fa5d963e3b1870ad81723a35f466e6966c14"}
Sep 08 18:52:03.433263 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.433171    2579 scope.go:117] "RemoveContainer" containerID="a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30"
Sep 08 18:52:03.433664 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:52:03.433637    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30\": container with ID starting with a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30 not found: ID does not exist" containerID="a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30"
Sep 08 18:52:03.433769 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.433674    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30"} err="failed to get container status \"a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30\": rpc error: code = NotFound desc = could not find container \"a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30\": container with ID starting with a8a2083d75b344694e1ae44fc09df0f875fc2fae1b9c1a09954adf94e84efa30 not found: ID does not exist"
Sep 08 18:52:03.436377 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:52:03.436351    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7704-d3a248\" is forbidden: User \"system:node:ip-10-0-140-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-26.ec2.internal' and this object" podUID="154fe63a-0f77-4eb2-b1bb-719413b622ef" pod="opendatahub/test-curl-7704-d3a248"
Sep 08 18:52:03.629154 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.629116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:03.731134 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:03.731051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:04.163849 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:04.163813    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="154fe63a-0f77-4eb2-b1bb-719413b622ef" path="/var/lib/kubelet/pods/154fe63a-0f77-4eb2-b1bb-719413b622ef/volumes"
Sep 08 18:52:04.434114 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:04.434032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d" event={"ID":"405a395a-5026-45e4-bd48-f921f6ef7885","Type":"ContainerStarted","Data":"85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555"}
Sep 08 18:52:04.434271 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:04.434131    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:04.460186 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:04.460031    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d" podStartSLOduration=1.8789569959999999 podStartE2EDuration="3.459995833s" totalImagesPullingTime="1.581038837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:04.458253496 +0000 UTC m=+1084.812346831" watchObservedRunningTime="2026-09-08 18:52:04.459995833 +0000 UTC m=+1084.814089168"
Sep 08 18:52:05.150245 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.150216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:05.245913 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.245880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:05.642425 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.642389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:05.723567 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.723530    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:52:05.744174 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.744143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:05.925139 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.925062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:05.964976 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.964947    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"]
Sep 08 18:52:05.968415 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.968398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:05.972075 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.972054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-235f65-dockercfg-rph7h\""
Sep 08 18:52:05.972205 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.972135    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-235f65\""
Sep 08 18:52:05.995487 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:05.995463    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"]
Sep 08 18:52:06.111607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.111575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmvh\" (UniqueName: \"kubernetes.io/projected/a2f336db-f177-4aa6-bb1f-4262186b8b88-kube-api-access-tsmvh\") pod \"payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5\" (UID: \"a2f336db-f177-4aa6-bb1f-4262186b8b88\") " pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:06.111791 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.111618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f336db-f177-4aa6-bb1f-4262186b8b88-plugins-config-volume\") pod \"payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5\" (UID: \"a2f336db-f177-4aa6-bb1f-4262186b8b88\") " pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:06.212918 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.212744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmvh\" (UniqueName: \"kubernetes.io/projected/a2f336db-f177-4aa6-bb1f-4262186b8b88-kube-api-access-tsmvh\") pod \"payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5\" (UID: \"a2f336db-f177-4aa6-bb1f-4262186b8b88\") " pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:06.213095 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.212921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f336db-f177-4aa6-bb1f-4262186b8b88-plugins-config-volume\") pod \"payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5\" (UID: \"a2f336db-f177-4aa6-bb1f-4262186b8b88\") " pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:06.213634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.213611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f336db-f177-4aa6-bb1f-4262186b8b88-plugins-config-volume\") pod \"payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5\" (UID: \"a2f336db-f177-4aa6-bb1f-4262186b8b88\") " pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:06.224032 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.224003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmvh\" (UniqueName: \"kubernetes.io/projected/a2f336db-f177-4aa6-bb1f-4262186b8b88-kube-api-access-tsmvh\") pod \"payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5\" (UID: \"a2f336db-f177-4aa6-bb1f-4262186b8b88\") " pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:06.278698 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.278662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:06.363506 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.363474    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"]
Sep 08 18:52:06.368160 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.368137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.371749 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.371726    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-3edf322b-kserve-self-signed-certs\""
Sep 08 18:52:06.379674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.379653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"]
Sep 08 18:52:06.445500 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.445467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n" event={"ID":"09566ff2-c460-4ac1-b944-b6547debe960","Type":"ContainerStarted","Data":"05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85"}
Sep 08 18:52:06.445679 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.445597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:06.466872 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.466783    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n" podStartSLOduration=1.8520406550000001 podStartE2EDuration="4.466769262s" totalImagesPullingTime="2.614728607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:06.464502548 +0000 UTC m=+1086.818595881" watchObservedRunningTime="2026-09-08 18:52:06.466769262 +0000 UTC m=+1086.820862594"
Sep 08 18:52:06.516181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.516158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tmp-dir\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.516317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.516212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-model-cache\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.516317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.516243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8lh\" (UniqueName: \"kubernetes.io/projected/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-kube-api-access-9s8lh\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.516317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.516260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tls-certs\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.516317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.516289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-home\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.516487 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.516366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-dshm\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.557029 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.557004    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"]
Sep 08 18:52:06.559432 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:52:06.559402    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f336db_f177_4aa6_bb1f_4262186b8b88.slice/crio-da241dd8c645ff3b44f99dad10f99994f61818920582158aae1ddd9f12706b57 WatchSource:0}: Error finding container da241dd8c645ff3b44f99dad10f99994f61818920582158aae1ddd9f12706b57: Status 404 returned error can't find the container with id da241dd8c645ff3b44f99dad10f99994f61818920582158aae1ddd9f12706b57
Sep 08 18:52:06.617244 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-home\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.617359 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-dshm\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.617359 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tmp-dir\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.617481 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-model-cache\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.617481 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8lh\" (UniqueName: \"kubernetes.io/projected/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-kube-api-access-9s8lh\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.617589 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tls-certs\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.617642 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-home\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.617702 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tmp-dir\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.617890 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.617862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-model-cache\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.619677 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.619661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-dshm\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.619962 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.619942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tls-certs\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.625711 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.625691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8lh\" (UniqueName: \"kubernetes.io/projected/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-kube-api-access-9s8lh\") pod \"e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") " pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.678615 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.678582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:06.892843 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.892813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:06.966475 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:52:06.966444    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbaa7c56_81c2_4bc9_9dd5_de238eab8671.slice/crio-7729ca5a1e7e0e134407d356929484fc6c7c65c6653afa2a5fe055209e09ba2f WatchSource:0}: Error finding container 7729ca5a1e7e0e134407d356929484fc6c7c65c6653afa2a5fe055209e09ba2f: Status 404 returned error can't find the container with id 7729ca5a1e7e0e134407d356929484fc6c7c65c6653afa2a5fe055209e09ba2f
Sep 08 18:52:06.967214 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:06.967176    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"]
Sep 08 18:52:07.366089 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.366053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:07.453969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.453929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp" event={"ID":"fbaa7c56-81c2-4bc9-9dd5-de238eab8671","Type":"ContainerStarted","Data":"d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0"}
Sep 08 18:52:07.454173 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.453993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp" event={"ID":"fbaa7c56-81c2-4bc9-9dd5-de238eab8671","Type":"ContainerStarted","Data":"7729ca5a1e7e0e134407d356929484fc6c7c65c6653afa2a5fe055209e09ba2f"}
Sep 08 18:52:07.455652 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.455620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5" event={"ID":"a2f336db-f177-4aa6-bb1f-4262186b8b88","Type":"ContainerStarted","Data":"b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873"}
Sep 08 18:52:07.455782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.455679    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:07.455782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.455700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5" event={"ID":"a2f336db-f177-4aa6-bb1f-4262186b8b88","Type":"ContainerStarted","Data":"da241dd8c645ff3b44f99dad10f99994f61818920582158aae1ddd9f12706b57"}
Sep 08 18:52:07.477684 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.477626    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp" podStartSLOduration=1.477611517 podStartE2EDuration="1.477611517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:07.475316997 +0000 UTC m=+1087.829410337" watchObservedRunningTime="2026-09-08 18:52:07.477611517 +0000 UTC m=+1087.831704850"
Sep 08 18:52:07.495819 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.495771    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5" podStartSLOduration=2.495754678 podStartE2EDuration="2.495754678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:07.494741858 +0000 UTC m=+1087.848835206" watchObservedRunningTime="2026-09-08 18:52:07.495754678 +0000 UTC m=+1087.849848016"
Sep 08 18:52:07.541089 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.541056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:07.641313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.641233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:07.743018 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:07.742979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:10.445128 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:10.445097    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:10.644662 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:10.644630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:12.079809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:12.079778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:12.169905 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:12.169874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:12.343030 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:12.342933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:12.496256 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:12.496209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:14.497215 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:14.497180    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:14.605664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:14.605631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:14.725733 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:14.725696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:16.529980 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:16.529945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:16.679237 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:16.679202    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:16.679422 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:16.679246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:16.691771 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:16.691747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:17.024149 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:17.024116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:17.379857 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:17.379824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:17.457527 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:17.457498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:17.514906 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:17.514877    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:52:17.527822 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:17.527718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:18.463448 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:18.463417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:52:19.640046 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:19.640012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:19.710948 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:19.710908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:19.800849 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:19.800817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:20.013966 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:20.013881    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:20.303440 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:20.303327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:20.750664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:20.750628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:20.864497 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:20.864457    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:22.019615 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:22.019580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:22.180040 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:22.180004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:22.280473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:22.280384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:22.590738 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:22.590700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:22.693963 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:22.693935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:24.199630 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:24.199597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:25.783315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:25.783282    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:52:25.944886 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:25.944846    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:26.046103 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:26.046001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:27.758777 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:27.758740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:33.678671 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:33.678623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:42.012336 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.012301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:42.531512 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.531477    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"]
Sep 08 18:52:42.531815 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.531751    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d" podUID="405a395a-5026-45e4-bd48-f921f6ef7885" containerName="maas-api" containerID="cri-o://85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555" gracePeriod=30
Sep 08 18:52:42.614482 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.614442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:42.654414 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.654389    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"]
Sep 08 18:52:42.654607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.654584    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n" podUID="09566ff2-c460-4ac1-b944-b6547debe960" containerName="payload-pre-processing" containerID="cri-o://05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85" gracePeriod=30
Sep 08 18:52:42.728407 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.728327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:42.791102 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.791042    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:42.812435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.812400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:42.879432 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.879398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:42.912783 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.912759    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:42.945694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.945663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-metrics-certs\") pod \"405a395a-5026-45e4-bd48-f921f6ef7885\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") "
Sep 08 18:52:42.945887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.945863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t98kx\" (UniqueName: \"kubernetes.io/projected/405a395a-5026-45e4-bd48-f921f6ef7885-kube-api-access-t98kx\") pod \"405a395a-5026-45e4-bd48-f921f6ef7885\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") "
Sep 08 18:52:42.945974 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.945958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a395a-5026-45e4-bd48-f921f6ef7885-openshift-service-ca\") pod \"405a395a-5026-45e4-bd48-f921f6ef7885\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") "
Sep 08 18:52:42.946028 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.946004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-maas-api-tls\") pod \"405a395a-5026-45e4-bd48-f921f6ef7885\" (UID: \"405a395a-5026-45e4-bd48-f921f6ef7885\") "
Sep 08 18:52:42.946355 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.946312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405a395a-5026-45e4-bd48-f921f6ef7885-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "405a395a-5026-45e4-bd48-f921f6ef7885" (UID: "405a395a-5026-45e4-bd48-f921f6ef7885"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:52:42.947945 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.947918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405a395a-5026-45e4-bd48-f921f6ef7885-kube-api-access-t98kx" (OuterVolumeSpecName: "kube-api-access-t98kx") pod "405a395a-5026-45e4-bd48-f921f6ef7885" (UID: "405a395a-5026-45e4-bd48-f921f6ef7885"). InnerVolumeSpecName "kube-api-access-t98kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:52:42.947945 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.947932    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "405a395a-5026-45e4-bd48-f921f6ef7885" (UID: "405a395a-5026-45e4-bd48-f921f6ef7885"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:52:42.948073 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.947979    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "405a395a-5026-45e4-bd48-f921f6ef7885" (UID: "405a395a-5026-45e4-bd48-f921f6ef7885"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:52:42.995946 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:42.995916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:43.047505 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.047422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g9bq\" (UniqueName: \"kubernetes.io/projected/09566ff2-c460-4ac1-b944-b6547debe960-kube-api-access-8g9bq\") pod \"09566ff2-c460-4ac1-b944-b6547debe960\" (UID: \"09566ff2-c460-4ac1-b944-b6547debe960\") "
Sep 08 18:52:43.047872 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.047560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09566ff2-c460-4ac1-b944-b6547debe960-plugins-config-volume\") pod \"09566ff2-c460-4ac1-b944-b6547debe960\" (UID: \"09566ff2-c460-4ac1-b944-b6547debe960\") "
Sep 08 18:52:43.047872 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.047704    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98kx\" (UniqueName: \"kubernetes.io/projected/405a395a-5026-45e4-bd48-f921f6ef7885-kube-api-access-t98kx\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:52:43.047872 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.047715    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a395a-5026-45e4-bd48-f921f6ef7885-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:52:43.047872 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.047725    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:52:43.047872 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.047734    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405a395a-5026-45e4-bd48-f921f6ef7885-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:52:43.048037 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.047876    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09566ff2-c460-4ac1-b944-b6547debe960-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09566ff2-c460-4ac1-b944-b6547debe960" (UID: "09566ff2-c460-4ac1-b944-b6547debe960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:52:43.049660 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.049643    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09566ff2-c460-4ac1-b944-b6547debe960-kube-api-access-8g9bq" (OuterVolumeSpecName: "kube-api-access-8g9bq") pod "09566ff2-c460-4ac1-b944-b6547debe960" (UID: "09566ff2-c460-4ac1-b944-b6547debe960"). InnerVolumeSpecName "kube-api-access-8g9bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:52:43.093470 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.093439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:43.148172 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.148150    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09566ff2-c460-4ac1-b944-b6547debe960-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:52:43.148172 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.148170    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g9bq\" (UniqueName: \"kubernetes.io/projected/09566ff2-c460-4ac1-b944-b6547debe960-kube-api-access-8g9bq\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:52:43.510741 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.510712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:43.605376 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.605334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:43.635333 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.635295    2579 generic.go:356] "Generic (PLEG): container finished" podID="405a395a-5026-45e4-bd48-f921f6ef7885" containerID="85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555" exitCode=0
Sep 08 18:52:43.635536 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.635413    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"
Sep 08 18:52:43.635536 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.635443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d" event={"ID":"405a395a-5026-45e4-bd48-f921f6ef7885","Type":"ContainerDied","Data":"85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555"}
Sep 08 18:52:43.635536 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.635512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d" event={"ID":"405a395a-5026-45e4-bd48-f921f6ef7885","Type":"ContainerDied","Data":"b73b5f64542c4cd8f7232a9971f6b267067477370f2bf34444316e713c070b07"}
Sep 08 18:52:43.635678 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.635551    2579 scope.go:117] "RemoveContainer" containerID="85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555"
Sep 08 18:52:43.637186 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.636930    2579 generic.go:356] "Generic (PLEG): container finished" podID="09566ff2-c460-4ac1-b944-b6547debe960" containerID="05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85" exitCode=0
Sep 08 18:52:43.637186 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.636972    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"
Sep 08 18:52:43.637186 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.637006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n" event={"ID":"09566ff2-c460-4ac1-b944-b6547debe960","Type":"ContainerDied","Data":"05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85"}
Sep 08 18:52:43.637186 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.637055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n" event={"ID":"09566ff2-c460-4ac1-b944-b6547debe960","Type":"ContainerDied","Data":"a7cd4f457406bd62af5770446358fa5d963e3b1870ad81723a35f466e6966c14"}
Sep 08 18:52:43.645643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.645492    2579 scope.go:117] "RemoveContainer" containerID="85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555"
Sep 08 18:52:43.645927 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:52:43.645897    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555\": container with ID starting with 85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555 not found: ID does not exist" containerID="85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555"
Sep 08 18:52:43.646038 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.645937    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555"} err="failed to get container status \"85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555\": rpc error: code = NotFound desc = could not find container \"85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555\": container with ID starting with 85cbce316db7b5d3a2a6e67d7aa1c7dfb4e346a8f831d1b2d12619a81d7f9555 not found: ID does not exist"
Sep 08 18:52:43.646038 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.645964    2579 scope.go:117] "RemoveContainer" containerID="05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85"
Sep 08 18:52:43.653980 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.653964    2579 scope.go:117] "RemoveContainer" containerID="05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85"
Sep 08 18:52:43.654220 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:52:43.654204    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85\": container with ID starting with 05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85 not found: ID does not exist" containerID="05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85"
Sep 08 18:52:43.654336 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.654226    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85"} err="failed to get container status \"05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85\": rpc error: code = NotFound desc = could not find container \"05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85\": container with ID starting with 05016dcb86b9d57f40b1e4755d021371bd337bbc3187ca4ea4ddec633424aa85 not found: ID does not exist"
Sep 08 18:52:43.660976 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.660954    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"]
Sep 08 18:52:43.667264 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.667244    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffda9270-6fc9b7d7cd-j8s6n"]
Sep 08 18:52:43.678206 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.678182    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"]
Sep 08 18:52:43.682918 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.682895    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffda9270-686974c47f-fmc8d"]
Sep 08 18:52:43.719655 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:43.719616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:44.041315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:44.041270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:44.166607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:44.166572    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09566ff2-c460-4ac1-b944-b6547debe960" path="/var/lib/kubelet/pods/09566ff2-c460-4ac1-b944-b6547debe960/volumes"
Sep 08 18:52:44.166976 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:44.166955    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="405a395a-5026-45e4-bd48-f921f6ef7885" path="/var/lib/kubelet/pods/405a395a-5026-45e4-bd48-f921f6ef7885/volumes"
Sep 08 18:52:44.259955 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:44.259902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:45.469843 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:45.469807    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:52:45.792537 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:45.792453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:47.057248 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.057215    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:47.147043 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.147016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:47.288389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.288360    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"]
Sep 08 18:52:47.288735 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.288722    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405a395a-5026-45e4-bd48-f921f6ef7885" containerName="maas-api"
Sep 08 18:52:47.288788 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.288737    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="405a395a-5026-45e4-bd48-f921f6ef7885" containerName="maas-api"
Sep 08 18:52:47.288788 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.288754    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09566ff2-c460-4ac1-b944-b6547debe960" containerName="payload-pre-processing"
Sep 08 18:52:47.288788 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.288761    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09566ff2-c460-4ac1-b944-b6547debe960" containerName="payload-pre-processing"
Sep 08 18:52:47.288888 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.288818    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09566ff2-c460-4ac1-b944-b6547debe960" containerName="payload-pre-processing"
Sep 08 18:52:47.288888 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.288829    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="405a395a-5026-45e4-bd48-f921f6ef7885" containerName="maas-api"
Sep 08 18:52:47.293302 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.293270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.295369 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.295331    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-62eb1773-serving-cert\""
Sep 08 18:52:47.295572 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.295332    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:52:47.296498 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.296310    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 18:52:47.296632 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.296553    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:52:47.296694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.296629    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:52:47.302869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.302841    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"]
Sep 08 18:52:47.329689 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.329665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:47.392222 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.392194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533e9210-aff2-464e-9f46-874ec8eadbd3-openshift-service-ca\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.392444 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.392253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-metrics-certs\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.392444 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.392313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-maas-api-tls\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.392444 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.392334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngmpr\" (UniqueName: \"kubernetes.io/projected/533e9210-aff2-464e-9f46-874ec8eadbd3-kube-api-access-ngmpr\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.430141 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.430109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:47.493126 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.493081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533e9210-aff2-464e-9f46-874ec8eadbd3-openshift-service-ca\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.493322 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.493142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-metrics-certs\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.493322 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.493197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-maas-api-tls\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.493322 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.493237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngmpr\" (UniqueName: \"kubernetes.io/projected/533e9210-aff2-464e-9f46-874ec8eadbd3-kube-api-access-ngmpr\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.493782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.493762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533e9210-aff2-464e-9f46-874ec8eadbd3-openshift-service-ca\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.495652 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.495625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-metrics-certs\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.495782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.495663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-maas-api-tls\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.501499 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.501467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngmpr\" (UniqueName: \"kubernetes.io/projected/533e9210-aff2-464e-9f46-874ec8eadbd3-kube-api-access-ngmpr\") pod \"maas-api-e2e-migrate-62eb1773-f567857b-s5b6j\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.606533 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.606439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:47.803966 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.803938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:47.882059 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.882034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"]
Sep 08 18:52:47.884817 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:52:47.884787    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533e9210_aff2_464e_9f46_874ec8eadbd3.slice/crio-4f701621a4fa9b161e3e061af79247bd0a90757fb41155ec9403a34831a0be37 WatchSource:0}: Error finding container 4f701621a4fa9b161e3e061af79247bd0a90757fb41155ec9403a34831a0be37: Status 404 returned error can't find the container with id 4f701621a4fa9b161e3e061af79247bd0a90757fb41155ec9403a34831a0be37
Sep 08 18:52:47.893902 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.893878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:47.907902 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.907875    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"]
Sep 08 18:52:47.912616 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.912600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:47.914847 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.914821    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-62eb1773\""
Sep 08 18:52:47.922382 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.922363    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"]
Sep 08 18:52:47.949896 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.949859    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"]
Sep 08 18:52:47.953101 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.953087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:47.955088 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.955072    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-62eb1773-dockercfg-wxsgb\""
Sep 08 18:52:47.963714 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.963696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"]
Sep 08 18:52:47.997774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.997752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677a649-77fd-4f55-a108-4598ceebd4c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz\" (UID: \"b677a649-77fd-4f55-a108-4598ceebd4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:47.997900 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:47.997800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxhh\" (UniqueName: \"kubernetes.io/projected/b677a649-77fd-4f55-a108-4598ceebd4c9-kube-api-access-lsxhh\") pod \"payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz\" (UID: \"b677a649-77fd-4f55-a108-4598ceebd4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:48.098929 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.098883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677a649-77fd-4f55-a108-4598ceebd4c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz\" (UID: \"b677a649-77fd-4f55-a108-4598ceebd4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:48.099402 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.098954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxhh\" (UniqueName: \"kubernetes.io/projected/b677a649-77fd-4f55-a108-4598ceebd4c9-kube-api-access-lsxhh\") pod \"payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz\" (UID: \"b677a649-77fd-4f55-a108-4598ceebd4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:48.099402 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.098988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71387a0c-c61c-4d08-9949-73d05edd9d74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n\" (UID: \"71387a0c-c61c-4d08-9949-73d05edd9d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:48.099402 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.099020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbr4\" (UniqueName: \"kubernetes.io/projected/71387a0c-c61c-4d08-9949-73d05edd9d74-kube-api-access-6pbr4\") pod \"payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n\" (UID: \"71387a0c-c61c-4d08-9949-73d05edd9d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:48.099589 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.099571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677a649-77fd-4f55-a108-4598ceebd4c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz\" (UID: \"b677a649-77fd-4f55-a108-4598ceebd4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:48.107357 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.107323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxhh\" (UniqueName: \"kubernetes.io/projected/b677a649-77fd-4f55-a108-4598ceebd4c9-kube-api-access-lsxhh\") pod \"payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz\" (UID: \"b677a649-77fd-4f55-a108-4598ceebd4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:48.200019 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.199931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71387a0c-c61c-4d08-9949-73d05edd9d74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n\" (UID: \"71387a0c-c61c-4d08-9949-73d05edd9d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:48.200019 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.199984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbr4\" (UniqueName: \"kubernetes.io/projected/71387a0c-c61c-4d08-9949-73d05edd9d74-kube-api-access-6pbr4\") pod \"payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n\" (UID: \"71387a0c-c61c-4d08-9949-73d05edd9d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:48.200634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.200612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71387a0c-c61c-4d08-9949-73d05edd9d74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n\" (UID: \"71387a0c-c61c-4d08-9949-73d05edd9d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:48.207983 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.207958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbr4\" (UniqueName: \"kubernetes.io/projected/71387a0c-c61c-4d08-9949-73d05edd9d74-kube-api-access-6pbr4\") pod \"payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n\" (UID: \"71387a0c-c61c-4d08-9949-73d05edd9d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:48.223510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.223475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:48.262771 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.262740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:48.498657 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.498584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"]
Sep 08 18:52:48.501808 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:52:48.501770    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb677a649_77fd_4f55_a108_4598ceebd4c9.slice/crio-e06ce048cc31aa838beffef23004a7ff66034610a3ba42d783b1a63e508ef010 WatchSource:0}: Error finding container e06ce048cc31aa838beffef23004a7ff66034610a3ba42d783b1a63e508ef010: Status 404 returned error can't find the container with id e06ce048cc31aa838beffef23004a7ff66034610a3ba42d783b1a63e508ef010
Sep 08 18:52:48.546418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.546390    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"]
Sep 08 18:52:48.549688 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:52:48.549660    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71387a0c_c61c_4d08_9949_73d05edd9d74.slice/crio-c98f77ef041461f93f50be4adc0036afa22f33b2c587a0ede2fbde0cb00cef92 WatchSource:0}: Error finding container c98f77ef041461f93f50be4adc0036afa22f33b2c587a0ede2fbde0cb00cef92: Status 404 returned error can't find the container with id c98f77ef041461f93f50be4adc0036afa22f33b2c587a0ede2fbde0cb00cef92
Sep 08 18:52:48.668889 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.668853    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz" event={"ID":"b677a649-77fd-4f55-a108-4598ceebd4c9","Type":"ContainerStarted","Data":"358b9f568a878d4db94ccc934dde6ea90b9c709ba4fc7cd6fef86252040af7c0"}
Sep 08 18:52:48.669083 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.668918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz" event={"ID":"b677a649-77fd-4f55-a108-4598ceebd4c9","Type":"ContainerStarted","Data":"e06ce048cc31aa838beffef23004a7ff66034610a3ba42d783b1a63e508ef010"}
Sep 08 18:52:48.669083 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.669030    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:52:48.670833 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.670800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j" event={"ID":"533e9210-aff2-464e-9f46-874ec8eadbd3","Type":"ContainerStarted","Data":"7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1"}
Sep 08 18:52:48.670976 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.670855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j" event={"ID":"533e9210-aff2-464e-9f46-874ec8eadbd3","Type":"ContainerStarted","Data":"4f701621a4fa9b161e3e061af79247bd0a90757fb41155ec9403a34831a0be37"}
Sep 08 18:52:48.670976 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.670884    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:48.672750 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.672690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n" event={"ID":"71387a0c-c61c-4d08-9949-73d05edd9d74","Type":"ContainerStarted","Data":"5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f"}
Sep 08 18:52:48.672750 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.672735    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:48.672941 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.672756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n" event={"ID":"71387a0c-c61c-4d08-9949-73d05edd9d74","Type":"ContainerStarted","Data":"c98f77ef041461f93f50be4adc0036afa22f33b2c587a0ede2fbde0cb00cef92"}
Sep 08 18:52:48.687089 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.687044    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz" podStartSLOduration=1.687031303 podStartE2EDuration="1.687031303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:48.685630772 +0000 UTC m=+1129.039724110" watchObservedRunningTime="2026-09-08 18:52:48.687031303 +0000 UTC m=+1129.041124635"
Sep 08 18:52:48.702811 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.702750    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j" podStartSLOduration=1.281577152 podStartE2EDuration="1.702733416s" totalImagesPullingTime="421.156264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:48.702101849 +0000 UTC m=+1129.056195184" watchObservedRunningTime="2026-09-08 18:52:48.702733416 +0000 UTC m=+1129.056826752"
Sep 08 18:52:48.719892 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:48.719845    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n" podStartSLOduration=1.719830715 podStartE2EDuration="1.719830715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:48.717111958 +0000 UTC m=+1129.071205292" watchObservedRunningTime="2026-09-08 18:52:48.719830715 +0000 UTC m=+1129.073924049"
Sep 08 18:52:54.683839 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:54.683809    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:52:58.364113 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:58.364071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:58.464737 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:58.464699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:52:59.680662 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:59.680634    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:52:59.681119 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:52:59.680706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:53:01.981292 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:01.981249    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:53:17.894282 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:17.894248    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:53:18.472654 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.472623    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"]
Sep 08 18:53:18.472871 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.472847    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j" podUID="533e9210-aff2-464e-9f46-874ec8eadbd3" containerName="maas-api" containerID="cri-o://7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1" gracePeriod=30
Sep 08 18:53:18.505399 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.505367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:18.535224 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.535196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"]
Sep 08 18:53:18.535429 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.535407    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n" podUID="71387a0c-c61c-4d08-9949-73d05edd9d74" containerName="payload-processing" containerID="cri-o://5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f" gracePeriod=30
Sep 08 18:53:18.541248 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.541218    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"]
Sep 08 18:53:18.541566 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.541506    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz" podUID="b677a649-77fd-4f55-a108-4598ceebd4c9" containerName="payload-pre-processing" containerID="cri-o://358b9f568a878d4db94ccc934dde6ea90b9c709ba4fc7cd6fef86252040af7c0" gracePeriod=30
Sep 08 18:53:18.605665 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.605630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:18.706478 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.706445    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:18.764988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.764955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:18.806014 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.805988    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:53:18.808954 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.808935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:53:18.825874 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.825850    2579 generic.go:356] "Generic (PLEG): container finished" podID="b677a649-77fd-4f55-a108-4598ceebd4c9" containerID="358b9f568a878d4db94ccc934dde6ea90b9c709ba4fc7cd6fef86252040af7c0" exitCode=0
Sep 08 18:53:18.826010 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.825922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz" event={"ID":"b677a649-77fd-4f55-a108-4598ceebd4c9","Type":"ContainerDied","Data":"358b9f568a878d4db94ccc934dde6ea90b9c709ba4fc7cd6fef86252040af7c0"}
Sep 08 18:53:18.828213 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.827765    2579 generic.go:356] "Generic (PLEG): container finished" podID="533e9210-aff2-464e-9f46-874ec8eadbd3" containerID="7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1" exitCode=0
Sep 08 18:53:18.828213 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.827883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j" event={"ID":"533e9210-aff2-464e-9f46-874ec8eadbd3","Type":"ContainerDied","Data":"7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1"}
Sep 08 18:53:18.828213 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.827922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j" event={"ID":"533e9210-aff2-464e-9f46-874ec8eadbd3","Type":"ContainerDied","Data":"4f701621a4fa9b161e3e061af79247bd0a90757fb41155ec9403a34831a0be37"}
Sep 08 18:53:18.828213 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.827951    2579 scope.go:117] "RemoveContainer" containerID="7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1"
Sep 08 18:53:18.828213 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.828093    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"
Sep 08 18:53:18.830854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.830830    2579 generic.go:356] "Generic (PLEG): container finished" podID="71387a0c-c61c-4d08-9949-73d05edd9d74" containerID="5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f" exitCode=0
Sep 08 18:53:18.830969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.830867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n" event={"ID":"71387a0c-c61c-4d08-9949-73d05edd9d74","Type":"ContainerDied","Data":"5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f"}
Sep 08 18:53:18.830969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.830879    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"
Sep 08 18:53:18.830969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.830905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n" event={"ID":"71387a0c-c61c-4d08-9949-73d05edd9d74","Type":"ContainerDied","Data":"c98f77ef041461f93f50be4adc0036afa22f33b2c587a0ede2fbde0cb00cef92"}
Sep 08 18:53:18.840813 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.840793    2579 scope.go:117] "RemoveContainer" containerID="7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1"
Sep 08 18:53:18.841176 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:53:18.841147    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1\": container with ID starting with 7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1 not found: ID does not exist" containerID="7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1"
Sep 08 18:53:18.841275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.841191    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1"} err="failed to get container status \"7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1\": rpc error: code = NotFound desc = could not find container \"7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1\": container with ID starting with 7e93470a1a76b6e51561a39cc99f7b934e833459ea76f9e2ee861b710f1ca3c1 not found: ID does not exist"
Sep 08 18:53:18.841275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.841215    2579 scope.go:117] "RemoveContainer" containerID="5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f"
Sep 08 18:53:18.852679 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.852644    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:53:18.852799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.852780    2579 scope.go:117] "RemoveContainer" containerID="5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f"
Sep 08 18:53:18.853258 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:53:18.853228    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f\": container with ID starting with 5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f not found: ID does not exist" containerID="5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f"
Sep 08 18:53:18.853381 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.853259    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f"} err="failed to get container status \"5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f\": rpc error: code = NotFound desc = could not find container \"5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f\": container with ID starting with 5aeb3a6d1a7cf2a51c721ef2ef8e629a20aa223c04d86bc0c1c57bdb90927e7f not found: ID does not exist"
Sep 08 18:53:18.867041 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.867014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:18.887585 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.887561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71387a0c-c61c-4d08-9949-73d05edd9d74-plugins-config-volume\") pod \"71387a0c-c61c-4d08-9949-73d05edd9d74\" (UID: \"71387a0c-c61c-4d08-9949-73d05edd9d74\") "
Sep 08 18:53:18.887698 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.887682    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-metrics-certs\") pod \"533e9210-aff2-464e-9f46-874ec8eadbd3\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") "
Sep 08 18:53:18.887756 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.887739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pbr4\" (UniqueName: \"kubernetes.io/projected/71387a0c-c61c-4d08-9949-73d05edd9d74-kube-api-access-6pbr4\") pod \"71387a0c-c61c-4d08-9949-73d05edd9d74\" (UID: \"71387a0c-c61c-4d08-9949-73d05edd9d74\") "
Sep 08 18:53:18.887801 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.887774    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngmpr\" (UniqueName: \"kubernetes.io/projected/533e9210-aff2-464e-9f46-874ec8eadbd3-kube-api-access-ngmpr\") pod \"533e9210-aff2-464e-9f46-874ec8eadbd3\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") "
Sep 08 18:53:18.887859 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.887812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533e9210-aff2-464e-9f46-874ec8eadbd3-openshift-service-ca\") pod \"533e9210-aff2-464e-9f46-874ec8eadbd3\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") "
Sep 08 18:53:18.887859 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.887835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-maas-api-tls\") pod \"533e9210-aff2-464e-9f46-874ec8eadbd3\" (UID: \"533e9210-aff2-464e-9f46-874ec8eadbd3\") "
Sep 08 18:53:18.888234 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.888207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533e9210-aff2-464e-9f46-874ec8eadbd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "533e9210-aff2-464e-9f46-874ec8eadbd3" (UID: "533e9210-aff2-464e-9f46-874ec8eadbd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:18.888379 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.888330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71387a0c-c61c-4d08-9949-73d05edd9d74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71387a0c-c61c-4d08-9949-73d05edd9d74" (UID: "71387a0c-c61c-4d08-9949-73d05edd9d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:18.889832 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.889812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "533e9210-aff2-464e-9f46-874ec8eadbd3" (UID: "533e9210-aff2-464e-9f46-874ec8eadbd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:18.890322 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.890302    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71387a0c-c61c-4d08-9949-73d05edd9d74-kube-api-access-6pbr4" (OuterVolumeSpecName: "kube-api-access-6pbr4") pod "71387a0c-c61c-4d08-9949-73d05edd9d74" (UID: "71387a0c-c61c-4d08-9949-73d05edd9d74"). InnerVolumeSpecName "kube-api-access-6pbr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:18.890417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.890325    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533e9210-aff2-464e-9f46-874ec8eadbd3-kube-api-access-ngmpr" (OuterVolumeSpecName: "kube-api-access-ngmpr") pod "533e9210-aff2-464e-9f46-874ec8eadbd3" (UID: "533e9210-aff2-464e-9f46-874ec8eadbd3"). InnerVolumeSpecName "kube-api-access-ngmpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:18.890417 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.890382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "533e9210-aff2-464e-9f46-874ec8eadbd3" (UID: "533e9210-aff2-464e-9f46-874ec8eadbd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:18.988928 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.988840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsxhh\" (UniqueName: \"kubernetes.io/projected/b677a649-77fd-4f55-a108-4598ceebd4c9-kube-api-access-lsxhh\") pod \"b677a649-77fd-4f55-a108-4598ceebd4c9\" (UID: \"b677a649-77fd-4f55-a108-4598ceebd4c9\") "
Sep 08 18:53:18.989396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.988963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677a649-77fd-4f55-a108-4598ceebd4c9-plugins-config-volume\") pod \"b677a649-77fd-4f55-a108-4598ceebd4c9\" (UID: \"b677a649-77fd-4f55-a108-4598ceebd4c9\") "
Sep 08 18:53:18.989396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.989143    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71387a0c-c61c-4d08-9949-73d05edd9d74-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:18.989396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.989156    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:18.989396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.989165    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pbr4\" (UniqueName: \"kubernetes.io/projected/71387a0c-c61c-4d08-9949-73d05edd9d74-kube-api-access-6pbr4\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:18.989396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.989173    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngmpr\" (UniqueName: \"kubernetes.io/projected/533e9210-aff2-464e-9f46-874ec8eadbd3-kube-api-access-ngmpr\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:18.989396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.989181    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533e9210-aff2-464e-9f46-874ec8eadbd3-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:18.989396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.989190    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533e9210-aff2-464e-9f46-874ec8eadbd3-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:18.989396 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.989361    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b677a649-77fd-4f55-a108-4598ceebd4c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b677a649-77fd-4f55-a108-4598ceebd4c9" (UID: "b677a649-77fd-4f55-a108-4598ceebd4c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:18.990754 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:18.990734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b677a649-77fd-4f55-a108-4598ceebd4c9-kube-api-access-lsxhh" (OuterVolumeSpecName: "kube-api-access-lsxhh") pod "b677a649-77fd-4f55-a108-4598ceebd4c9" (UID: "b677a649-77fd-4f55-a108-4598ceebd4c9"). InnerVolumeSpecName "kube-api-access-lsxhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:19.090093 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.090049    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677a649-77fd-4f55-a108-4598ceebd4c9-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:19.090093 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.090089    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsxhh\" (UniqueName: \"kubernetes.io/projected/b677a649-77fd-4f55-a108-4598ceebd4c9-kube-api-access-lsxhh\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:19.150176 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.150145    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"]
Sep 08 18:53:19.156080 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.156058    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-62eb1773-f567857b-s5b6j"]
Sep 08 18:53:19.166184 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.166159    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"]
Sep 08 18:53:19.170349 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.170316    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-62eb1773-5cc66d9bf4-q8w6n"]
Sep 08 18:53:19.335012 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.334931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:19.669913 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.669810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:19.838126 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.838096    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"
Sep 08 18:53:19.838290 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.838100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz" event={"ID":"b677a649-77fd-4f55-a108-4598ceebd4c9","Type":"ContainerDied","Data":"e06ce048cc31aa838beffef23004a7ff66034610a3ba42d783b1a63e508ef010"}
Sep 08 18:53:19.838290 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.838212    2579 scope.go:117] "RemoveContainer" containerID="358b9f568a878d4db94ccc934dde6ea90b9c709ba4fc7cd6fef86252040af7c0"
Sep 08 18:53:19.858535 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.858508    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"]
Sep 08 18:53:19.863438 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:19.863413    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-62eb1773-77b9549758-djwtz"]
Sep 08 18:53:20.162509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:20.162473    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533e9210-aff2-464e-9f46-874ec8eadbd3" path="/var/lib/kubelet/pods/533e9210-aff2-464e-9f46-874ec8eadbd3/volumes"
Sep 08 18:53:20.162880 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:20.162837    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71387a0c-c61c-4d08-9949-73d05edd9d74" path="/var/lib/kubelet/pods/71387a0c-c61c-4d08-9949-73d05edd9d74/volumes"
Sep 08 18:53:20.163147 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:20.163133    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b677a649-77fd-4f55-a108-4598ceebd4c9" path="/var/lib/kubelet/pods/b677a649-77fd-4f55-a108-4598ceebd4c9/volumes"
Sep 08 18:53:20.352704 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:20.352653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:21.659709 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.659677    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"]
Sep 08 18:53:21.660086 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660043    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b677a649-77fd-4f55-a108-4598ceebd4c9" containerName="payload-pre-processing"
Sep 08 18:53:21.660086 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660079    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b677a649-77fd-4f55-a108-4598ceebd4c9" containerName="payload-pre-processing"
Sep 08 18:53:21.660163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660094    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533e9210-aff2-464e-9f46-874ec8eadbd3" containerName="maas-api"
Sep 08 18:53:21.660163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660100    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="533e9210-aff2-464e-9f46-874ec8eadbd3" containerName="maas-api"
Sep 08 18:53:21.660163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660109    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71387a0c-c61c-4d08-9949-73d05edd9d74" containerName="payload-processing"
Sep 08 18:53:21.660163 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660116    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71387a0c-c61c-4d08-9949-73d05edd9d74" containerName="payload-processing"
Sep 08 18:53:21.660288 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71387a0c-c61c-4d08-9949-73d05edd9d74" containerName="payload-processing"
Sep 08 18:53:21.660288 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660198    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b677a649-77fd-4f55-a108-4598ceebd4c9" containerName="payload-pre-processing"
Sep 08 18:53:21.660288 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.660207    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="533e9210-aff2-464e-9f46-874ec8eadbd3" containerName="maas-api"
Sep 08 18:53:21.664869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.664853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.667105 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.667088    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-3edf322b-kserve-self-signed-certs\""
Sep 08 18:53:21.674005 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.673984    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"]
Sep 08 18:53:21.814401 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.814366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-dshm\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.814619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.814432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-model-cache\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.814619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.814544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-home\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.814619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.814583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-tmp-dir\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.814619 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.814610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/f473666b-b923-4d9e-93a7-e8fe7a58636c-kube-api-access-m2gzd\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.814791 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.814653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f473666b-b923-4d9e-93a7-e8fe7a58636c-tls-certs\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.915735 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.915644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-model-cache\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.915735 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.915713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-home\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.915735 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.915735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-tmp-dir\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.916020 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.915753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/f473666b-b923-4d9e-93a7-e8fe7a58636c-kube-api-access-m2gzd\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.916020 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.915794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f473666b-b923-4d9e-93a7-e8fe7a58636c-tls-certs\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.916020 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.915852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-dshm\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.916165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.916078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-model-cache\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.916165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.916138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-home\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.916243 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.916173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-tmp-dir\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.918236 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.918212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-dshm\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.918442 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.918424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f473666b-b923-4d9e-93a7-e8fe7a58636c-tls-certs\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.923007 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.922982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/f473666b-b923-4d9e-93a7-e8fe7a58636c-kube-api-access-m2gzd\") pod \"e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") " pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.973700 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.973627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:21.979625 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:21.979598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:22.112263 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:22.112227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:22.198772 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:22.198691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:22.250818 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:22.250795    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"]
Sep 08 18:53:22.252659 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:53:22.252637    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf473666b_b923_4d9e_93a7_e8fe7a58636c.slice/crio-a0cb4872cd49c9e52240c42b3c0ef0b819401472509e4cc54d21733cc0440888 WatchSource:0}: Error finding container a0cb4872cd49c9e52240c42b3c0ef0b819401472509e4cc54d21733cc0440888: Status 404 returned error can't find the container with id a0cb4872cd49c9e52240c42b3c0ef0b819401472509e4cc54d21733cc0440888
Sep 08 18:53:22.856142 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:22.856107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6" event={"ID":"f473666b-b923-4d9e-93a7-e8fe7a58636c","Type":"ContainerStarted","Data":"d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70"}
Sep 08 18:53:22.856142 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:22.856148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6" event={"ID":"f473666b-b923-4d9e-93a7-e8fe7a58636c","Type":"ContainerStarted","Data":"a0cb4872cd49c9e52240c42b3c0ef0b819401472509e4cc54d21733cc0440888"}
Sep 08 18:53:22.875072 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:22.875019    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6" podStartSLOduration=1.8750038500000001 podStartE2EDuration="1.87500385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:22.874067191 +0000 UTC m=+1163.228160523" watchObservedRunningTime="2026-09-08 18:53:22.87500385 +0000 UTC m=+1163.229097183"
Sep 08 18:53:23.206854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.206778    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"]
Sep 08 18:53:23.211492 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.211468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.213502 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.213478    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a6830f31-serving-cert\""
Sep 08 18:53:23.214149 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.214118    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:53:23.214273 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.214119    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:53:23.214851 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.214126    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 18:53:23.214998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.214177    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:53:23.220087 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.220064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"]
Sep 08 18:53:23.243612 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.243571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:23.327993 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.327964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.328204 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.328004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw95t\" (UniqueName: \"kubernetes.io/projected/c3912e07-e008-4d21-a71a-d099e016688b-kube-api-access-gw95t\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.328204 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.328034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-metrics-certs\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.328204 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.328151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3912e07-e008-4d21-a71a-d099e016688b-openshift-service-ca\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.346294 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.346267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:23.428902 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.428876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3912e07-e008-4d21-a71a-d099e016688b-openshift-service-ca\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.429023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.428923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.429023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.428946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw95t\" (UniqueName: \"kubernetes.io/projected/c3912e07-e008-4d21-a71a-d099e016688b-kube-api-access-gw95t\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.429023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.428979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-metrics-certs\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.429181 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:53:23.429039    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-serving-cert: secret "maas-api-e2e-ait-a6830f31-serving-cert" not found
Sep 08 18:53:23.429181 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:53:23.429113    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls podName:c3912e07-e008-4d21-a71a-d099e016688b nodeName:}" failed. No retries permitted until 2026-09-08 18:53:23.929091391 +0000 UTC m=+1164.283184719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls") pod "maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv" (UID: "c3912e07-e008-4d21-a71a-d099e016688b") : secret "maas-api-e2e-ait-a6830f31-serving-cert" not found
Sep 08 18:53:23.429669 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.429651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3912e07-e008-4d21-a71a-d099e016688b-openshift-service-ca\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.431605 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.431586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-metrics-certs\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.439116 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.439085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw95t\" (UniqueName: \"kubernetes.io/projected/c3912e07-e008-4d21-a71a-d099e016688b-kube-api-access-gw95t\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.706982 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.706941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:23.801510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.801476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:23.860860 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.860824    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"]
Sep 08 18:53:23.864733 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.864713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:23.866694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.866670    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a6830f31-dockercfg-lpgm2\""
Sep 08 18:53:23.867037 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.867019    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a6830f31\""
Sep 08 18:53:23.878259 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.878236    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"]
Sep 08 18:53:23.886740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.883574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"]
Sep 08 18:53:23.886740 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.883683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:23.898004 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.897977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"]
Sep 08 18:53:23.914039 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.914008    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:23.934321 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.934293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs\" (UID: \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:23.934451 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.934399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnxj\" (UniqueName: \"kubernetes.io/projected/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-kube-api-access-ppnxj\") pod \"payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs\" (UID: \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:23.934499 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.934483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:23.936783 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:23.936757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls\") pod \"maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:24.036043 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.035939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx\" (UID: \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\") " pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:24.036043 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.036023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs\" (UID: \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:24.036262 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.036064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrcn\" (UniqueName: \"kubernetes.io/projected/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-kube-api-access-zlrcn\") pod \"payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx\" (UID: \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\") " pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:24.036262 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.036104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnxj\" (UniqueName: \"kubernetes.io/projected/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-kube-api-access-ppnxj\") pod \"payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs\" (UID: \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:24.036755 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.036722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs\" (UID: \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:24.043699 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.043674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnxj\" (UniqueName: \"kubernetes.io/projected/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-kube-api-access-ppnxj\") pod \"payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs\" (UID: \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:24.124816 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.124784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:24.129180 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.129146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:24.137374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.137323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx\" (UID: \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\") " pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:24.137504 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.137434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrcn\" (UniqueName: \"kubernetes.io/projected/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-kube-api-access-zlrcn\") pod \"payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx\" (UID: \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\") " pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:24.137991 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.137969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx\" (UID: \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\") " pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:24.146598 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.146574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrcn\" (UniqueName: \"kubernetes.io/projected/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-kube-api-access-zlrcn\") pod \"payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx\" (UID: \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\") " pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:24.174995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.174751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:24.196691 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.196668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:24.397301 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.397274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"]
Sep 08 18:53:24.399813 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:53:24.399791    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3912e07_e008_4d21_a71a_d099e016688b.slice/crio-c3aa267531ffa062245fa661301dc35856f60a6d65a853d7ae7ca2e11d3539d1 WatchSource:0}: Error finding container c3aa267531ffa062245fa661301dc35856f60a6d65a853d7ae7ca2e11d3539d1: Status 404 returned error can't find the container with id c3aa267531ffa062245fa661301dc35856f60a6d65a853d7ae7ca2e11d3539d1
Sep 08 18:53:24.452023 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.452000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"]
Sep 08 18:53:24.453658 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:53:24.453632    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0923e1_bdd9_4267_bafd_32b864a2aa3d.slice/crio-7d6a47fe724cd962814decd32a1304e239a23273b8eb8a2b67b9a7c7371f1d3c WatchSource:0}: Error finding container 7d6a47fe724cd962814decd32a1304e239a23273b8eb8a2b67b9a7c7371f1d3c: Status 404 returned error can't find the container with id 7d6a47fe724cd962814decd32a1304e239a23273b8eb8a2b67b9a7c7371f1d3c
Sep 08 18:53:24.477874 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.477854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"]
Sep 08 18:53:24.483454 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:53:24.483426    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b03dda_1da8_4f15_9583_e91e9fa8bb4c.slice/crio-cf2f14fc1e34f170cfa76265a66db4d5aaeca809019ed67731ee0968d6455a78 WatchSource:0}: Error finding container cf2f14fc1e34f170cfa76265a66db4d5aaeca809019ed67731ee0968d6455a78: Status 404 returned error can't find the container with id cf2f14fc1e34f170cfa76265a66db4d5aaeca809019ed67731ee0968d6455a78
Sep 08 18:53:24.869162 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.869124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv" event={"ID":"c3912e07-e008-4d21-a71a-d099e016688b","Type":"ContainerStarted","Data":"c3aa267531ffa062245fa661301dc35856f60a6d65a853d7ae7ca2e11d3539d1"}
Sep 08 18:53:24.870661 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.870631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx" event={"ID":"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c","Type":"ContainerStarted","Data":"0c91fc124fba129bbbca29debe3e5bcb8379f32e14e0917e803675e54bec43d5"}
Sep 08 18:53:24.870809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.870672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx" event={"ID":"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c","Type":"ContainerStarted","Data":"cf2f14fc1e34f170cfa76265a66db4d5aaeca809019ed67731ee0968d6455a78"}
Sep 08 18:53:24.870809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.870752    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:24.872324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.872294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs" event={"ID":"8c0923e1-bdd9-4267-bafd-32b864a2aa3d","Type":"ContainerStarted","Data":"7d5bad2dd4a946d8098278be62ae958e235c3554d861e59a6a5d7dd094e4f9d5"}
Sep 08 18:53:24.872427 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.872328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs" event={"ID":"8c0923e1-bdd9-4267-bafd-32b864a2aa3d","Type":"ContainerStarted","Data":"7d6a47fe724cd962814decd32a1304e239a23273b8eb8a2b67b9a7c7371f1d3c"}
Sep 08 18:53:24.872427 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.872374    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:24.887444 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.887405    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx" podStartSLOduration=1.8873936279999999 podStartE2EDuration="1.887393628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:24.88616867 +0000 UTC m=+1165.240262004" watchObservedRunningTime="2026-09-08 18:53:24.887393628 +0000 UTC m=+1165.241486960"
Sep 08 18:53:24.902727 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:24.902685    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs" podStartSLOduration=1.902676967 podStartE2EDuration="1.902676967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:24.900275426 +0000 UTC m=+1165.254368758" watchObservedRunningTime="2026-09-08 18:53:24.902676967 +0000 UTC m=+1165.256770299"
Sep 08 18:53:25.879011 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:25.878964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv" event={"ID":"c3912e07-e008-4d21-a71a-d099e016688b","Type":"ContainerStarted","Data":"3cfcf876b08ef8a304ad16a126dc7806a349314a81a934b447cfd2db7eddb037"}
Sep 08 18:53:25.879482 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:25.879358    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:25.898749 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:25.898700    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv" podStartSLOduration=2.354616941 podStartE2EDuration="2.898681111s" totalImagesPullingTime="544.06417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:25.897523179 +0000 UTC m=+1166.251616517" watchObservedRunningTime="2026-09-08 18:53:25.898681111 +0000 UTC m=+1166.252774446"
Sep 08 18:53:31.889803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:31.889774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:31.974428 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:31.974397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:31.974589 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:31.974461    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:31.986743 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:31.986714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:32.924657 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:32.924626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:53:35.881311 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:35.881273    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:35.881778 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:35.881465    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:37.374478 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:37.374444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:53:42.474727 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.474693    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"]
Sep 08 18:53:42.478436 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.478418    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.481150 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.481126    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-235f65\"/\"autores-model-235f65-kserve-self-signed-certs\""
Sep 08 18:53:42.481254 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.481126    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-235f65\"/\"kube-root-ca.crt\""
Sep 08 18:53:42.481254 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.481173    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-235f65\"/\"openshift-service-ca.crt\""
Sep 08 18:53:42.481254 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.481127    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-235f65\"/\"default-dockercfg-d7k2l\""
Sep 08 18:53:42.487462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.487439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"]
Sep 08 18:53:42.509141 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.509114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-tmp-dir\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.509281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.509147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zcf8\" (UniqueName: \"kubernetes.io/projected/3e614ead-0f34-4df6-828c-baa444ff648f-kube-api-access-4zcf8\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.509281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.509180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-model-cache\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.509281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.509235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e614ead-0f34-4df6-828c-baa444ff648f-tls-certs\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.509281 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.509259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-home\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.509504 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.509417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-dshm\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.581480 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.581451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:42.610594 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.610542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-dshm\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.610594 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.610605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-tmp-dir\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.610790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.610626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zcf8\" (UniqueName: \"kubernetes.io/projected/3e614ead-0f34-4df6-828c-baa444ff648f-kube-api-access-4zcf8\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.610790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.610654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-model-cache\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.610790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.610684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e614ead-0f34-4df6-828c-baa444ff648f-tls-certs\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.610790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.610703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-home\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.611052 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.611026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-model-cache\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.611101 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.611070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-tmp-dir\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.611157 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.611138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-home\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.612973 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.612946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-dshm\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.613109 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.613090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e614ead-0f34-4df6-828c-baa444ff648f-tls-certs\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.617713 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.617695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zcf8\" (UniqueName: \"kubernetes.io/projected/3e614ead-0f34-4df6-828c-baa444ff648f-kube-api-access-4zcf8\") pod \"autores-model-235f65-kserve-69455d8779-tdt8v\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") " pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:42.789145 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:42.789055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:43.058647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:43.058620    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"]
Sep 08 18:53:43.060695 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:53:43.060664    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e614ead_0f34_4df6_828c_baa444ff648f.slice/crio-9b0e785c6492fc722dc7dba696cbd5d697b368f7d219df1930798885168dfc43 WatchSource:0}: Error finding container 9b0e785c6492fc722dc7dba696cbd5d697b368f7d219df1930798885168dfc43: Status 404 returned error can't find the container with id 9b0e785c6492fc722dc7dba696cbd5d697b368f7d219df1930798885168dfc43
Sep 08 18:53:43.969510 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:43.969469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v" event={"ID":"3e614ead-0f34-4df6-828c-baa444ff648f","Type":"ContainerStarted","Data":"7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c"}
Sep 08 18:53:43.969882 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:43.969519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v" event={"ID":"3e614ead-0f34-4df6-828c-baa444ff648f","Type":"ContainerStarted","Data":"9b0e785c6492fc722dc7dba696cbd5d697b368f7d219df1930798885168dfc43"}
Sep 08 18:53:43.984489 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:43.984438    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v" podStartSLOduration=1.984423376 podStartE2EDuration="1.984423376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:43.983302267 +0000 UTC m=+1184.337395599" watchObservedRunningTime="2026-09-08 18:53:43.984423376 +0000 UTC m=+1184.338516708"
Sep 08 18:53:52.789280 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:52.789245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:52.789280 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:52.789291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:52.801805 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:52.801778    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:53.023437 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.023407    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:53:53.782476 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.782443    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"]
Sep 08 18:53:53.782672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.782636    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv" podUID="c3912e07-e008-4d21-a71a-d099e016688b" containerName="maas-api" containerID="cri-o://3cfcf876b08ef8a304ad16a126dc7806a349314a81a934b447cfd2db7eddb037" gracePeriod=30
Sep 08 18:53:53.806959 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.806928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:53.834509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.834472    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"]
Sep 08 18:53:53.835008 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.834974    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx" podUID="e8b03dda-1da8-4f15-9583-e91e9fa8bb4c" containerName="payload-processing" containerID="cri-o://0c91fc124fba129bbbca29debe3e5bcb8379f32e14e0917e803675e54bec43d5" gracePeriod=30
Sep 08 18:53:53.838102 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.838059    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"]
Sep 08 18:53:53.838300 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.838275    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs" podUID="8c0923e1-bdd9-4267-bafd-32b864a2aa3d" containerName="payload-pre-processing" containerID="cri-o://7d5bad2dd4a946d8098278be62ae958e235c3554d861e59a6a5d7dd094e4f9d5" gracePeriod=30
Sep 08 18:53:53.866597 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.866567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:53.970413 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:53.970373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:54.019789 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.019760    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c0923e1-bdd9-4267-bafd-32b864a2aa3d" containerID="7d5bad2dd4a946d8098278be62ae958e235c3554d861e59a6a5d7dd094e4f9d5" exitCode=0
Sep 08 18:53:54.019945 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.019911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs" event={"ID":"8c0923e1-bdd9-4267-bafd-32b864a2aa3d","Type":"ContainerDied","Data":"7d5bad2dd4a946d8098278be62ae958e235c3554d861e59a6a5d7dd094e4f9d5"}
Sep 08 18:53:54.024135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.021497    2579 generic.go:356] "Generic (PLEG): container finished" podID="c3912e07-e008-4d21-a71a-d099e016688b" containerID="3cfcf876b08ef8a304ad16a126dc7806a349314a81a934b447cfd2db7eddb037" exitCode=0
Sep 08 18:53:54.024135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.021564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv" event={"ID":"c3912e07-e008-4d21-a71a-d099e016688b","Type":"ContainerDied","Data":"3cfcf876b08ef8a304ad16a126dc7806a349314a81a934b447cfd2db7eddb037"}
Sep 08 18:53:54.024135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.023033    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8b03dda-1da8-4f15-9583-e91e9fa8bb4c" containerID="0c91fc124fba129bbbca29debe3e5bcb8379f32e14e0917e803675e54bec43d5" exitCode=0
Sep 08 18:53:54.024135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.023106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx" event={"ID":"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c","Type":"ContainerDied","Data":"0c91fc124fba129bbbca29debe3e5bcb8379f32e14e0917e803675e54bec43d5"}
Sep 08 18:53:54.067636 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.067599    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:54.077398 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.077330    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:54.109515 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.109487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:54.131119 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.131090    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3912e07-e008-4d21-a71a-d099e016688b-openshift-service-ca\") pod \"c3912e07-e008-4d21-a71a-d099e016688b\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") "
Sep 08 18:53:54.131273 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.131196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-metrics-certs\") pod \"c3912e07-e008-4d21-a71a-d099e016688b\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") "
Sep 08 18:53:54.131273 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.131263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls\") pod \"c3912e07-e008-4d21-a71a-d099e016688b\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") "
Sep 08 18:53:54.131431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.131331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw95t\" (UniqueName: \"kubernetes.io/projected/c3912e07-e008-4d21-a71a-d099e016688b-kube-api-access-gw95t\") pod \"c3912e07-e008-4d21-a71a-d099e016688b\" (UID: \"c3912e07-e008-4d21-a71a-d099e016688b\") "
Sep 08 18:53:54.131498 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.131475    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3912e07-e008-4d21-a71a-d099e016688b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3912e07-e008-4d21-a71a-d099e016688b" (UID: "c3912e07-e008-4d21-a71a-d099e016688b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:54.131729 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.131706    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3912e07-e008-4d21-a71a-d099e016688b-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.132389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.132357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:54.133572 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.133550    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3912e07-e008-4d21-a71a-d099e016688b" (UID: "c3912e07-e008-4d21-a71a-d099e016688b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:54.133757 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.133730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3912e07-e008-4d21-a71a-d099e016688b-kube-api-access-gw95t" (OuterVolumeSpecName: "kube-api-access-gw95t") pod "c3912e07-e008-4d21-a71a-d099e016688b" (UID: "c3912e07-e008-4d21-a71a-d099e016688b"). InnerVolumeSpecName "kube-api-access-gw95t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:54.133879 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.133861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c3912e07-e008-4d21-a71a-d099e016688b" (UID: "c3912e07-e008-4d21-a71a-d099e016688b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:54.144947 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.144921    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:54.232674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.232644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-plugins-config-volume\") pod \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\" (UID: \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\") "
Sep 08 18:53:54.232845 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.232698    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-plugins-config-volume\") pod \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\" (UID: \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\") "
Sep 08 18:53:54.232845 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.232746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppnxj\" (UniqueName: \"kubernetes.io/projected/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-kube-api-access-ppnxj\") pod \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\" (UID: \"8c0923e1-bdd9-4267-bafd-32b864a2aa3d\") "
Sep 08 18:53:54.232845 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.232837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlrcn\" (UniqueName: \"kubernetes.io/projected/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-kube-api-access-zlrcn\") pod \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\" (UID: \"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c\") "
Sep 08 18:53:54.233076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.233043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c0923e1-bdd9-4267-bafd-32b864a2aa3d" (UID: "8c0923e1-bdd9-4267-bafd-32b864a2aa3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:54.233214 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.233189    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw95t\" (UniqueName: \"kubernetes.io/projected/c3912e07-e008-4d21-a71a-d099e016688b-kube-api-access-gw95t\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.233214 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.233105    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8b03dda-1da8-4f15-9583-e91e9fa8bb4c" (UID: "e8b03dda-1da8-4f15-9583-e91e9fa8bb4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:54.233214 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.233212    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.233400 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.233225    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3912e07-e008-4d21-a71a-d099e016688b-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.234912 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.234891    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-kube-api-access-ppnxj" (OuterVolumeSpecName: "kube-api-access-ppnxj") pod "8c0923e1-bdd9-4267-bafd-32b864a2aa3d" (UID: "8c0923e1-bdd9-4267-bafd-32b864a2aa3d"). InnerVolumeSpecName "kube-api-access-ppnxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:54.234982 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.234947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-kube-api-access-zlrcn" (OuterVolumeSpecName: "kube-api-access-zlrcn") pod "e8b03dda-1da8-4f15-9583-e91e9fa8bb4c" (UID: "e8b03dda-1da8-4f15-9583-e91e9fa8bb4c"). InnerVolumeSpecName "kube-api-access-zlrcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:54.334730 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.334699    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrcn\" (UniqueName: \"kubernetes.io/projected/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-kube-api-access-zlrcn\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.334730 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.334727    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.334918 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.334737    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.334918 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:54.334746    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppnxj\" (UniqueName: \"kubernetes.io/projected/8c0923e1-bdd9-4267-bafd-32b864a2aa3d-kube-api-access-ppnxj\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:53:55.029286 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.029255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"
Sep 08 18:53:55.029769 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.029288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv" event={"ID":"c3912e07-e008-4d21-a71a-d099e016688b","Type":"ContainerDied","Data":"c3aa267531ffa062245fa661301dc35856f60a6d65a853d7ae7ca2e11d3539d1"}
Sep 08 18:53:55.029769 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.029375    2579 scope.go:117] "RemoveContainer" containerID="3cfcf876b08ef8a304ad16a126dc7806a349314a81a934b447cfd2db7eddb037"
Sep 08 18:53:55.030768 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.030654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx" event={"ID":"e8b03dda-1da8-4f15-9583-e91e9fa8bb4c","Type":"ContainerDied","Data":"cf2f14fc1e34f170cfa76265a66db4d5aaeca809019ed67731ee0968d6455a78"}
Sep 08 18:53:55.030768 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.030680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"
Sep 08 18:53:55.032304 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.032280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs" event={"ID":"8c0923e1-bdd9-4267-bafd-32b864a2aa3d","Type":"ContainerDied","Data":"7d6a47fe724cd962814decd32a1304e239a23273b8eb8a2b67b9a7c7371f1d3c"}
Sep 08 18:53:55.032445 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.032292    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"
Sep 08 18:53:55.039743 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.039676    2579 scope.go:117] "RemoveContainer" containerID="0c91fc124fba129bbbca29debe3e5bcb8379f32e14e0917e803675e54bec43d5"
Sep 08 18:53:55.047742 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.047727    2579 scope.go:117] "RemoveContainer" containerID="7d5bad2dd4a946d8098278be62ae958e235c3554d861e59a6a5d7dd094e4f9d5"
Sep 08 18:53:55.056870 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.056850    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"]
Sep 08 18:53:55.062583 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.062563    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6830f31-58cb4bb7d-xtbzv"]
Sep 08 18:53:55.077778 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.077758    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"]
Sep 08 18:53:55.083119 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.083101    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6830f31-7cfc7449cf-ntzpx"]
Sep 08 18:53:55.095865 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.095847    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"]
Sep 08 18:53:55.100387 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.100368    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6830f31-6547848d94-2pnxs"]
Sep 08 18:53:55.260490 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.260452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:55.710954 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:55.710925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:56.089038 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:56.089007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:56.163184 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:56.163153    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0923e1-bdd9-4267-bafd-32b864a2aa3d" path="/var/lib/kubelet/pods/8c0923e1-bdd9-4267-bafd-32b864a2aa3d/volumes"
Sep 08 18:53:56.163592 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:56.163577    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3912e07-e008-4d21-a71a-d099e016688b" path="/var/lib/kubelet/pods/c3912e07-e008-4d21-a71a-d099e016688b/volumes"
Sep 08 18:53:56.163897 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:56.163885    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b03dda-1da8-4f15-9583-e91e9fa8bb4c" path="/var/lib/kubelet/pods/e8b03dda-1da8-4f15-9583-e91e9fa8bb4c/volumes"
Sep 08 18:53:56.178932 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:56.178903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:58.404467 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:58.404430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:58.492270 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:58.492240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:53:58.493681 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:58.493652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:59.509074 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509036    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"]
Sep 08 18:53:59.509474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509406    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0923e1-bdd9-4267-bafd-32b864a2aa3d" containerName="payload-pre-processing"
Sep 08 18:53:59.509474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509418    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0923e1-bdd9-4267-bafd-32b864a2aa3d" containerName="payload-pre-processing"
Sep 08 18:53:59.509474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509430    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b03dda-1da8-4f15-9583-e91e9fa8bb4c" containerName="payload-processing"
Sep 08 18:53:59.509474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509436    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b03dda-1da8-4f15-9583-e91e9fa8bb4c" containerName="payload-processing"
Sep 08 18:53:59.509474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509445    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3912e07-e008-4d21-a71a-d099e016688b" containerName="maas-api"
Sep 08 18:53:59.509474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509451    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3912e07-e008-4d21-a71a-d099e016688b" containerName="maas-api"
Sep 08 18:53:59.509694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509521    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b03dda-1da8-4f15-9583-e91e9fa8bb4c" containerName="payload-processing"
Sep 08 18:53:59.509694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509530    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0923e1-bdd9-4267-bafd-32b864a2aa3d" containerName="payload-pre-processing"
Sep 08 18:53:59.509694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.509536    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3912e07-e008-4d21-a71a-d099e016688b" containerName="maas-api"
Sep 08 18:53:59.512611 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.512589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.514757 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.514733    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:53:59.514757 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.514751    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:53:59.515367 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.515326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6956efcb-serving-cert\""
Sep 08 18:53:59.515470 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.515395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 18:53:59.515598 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.515576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:53:59.524366 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.524331    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"]
Sep 08 18:53:59.549791 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.549768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:59.586188 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.586162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-metrics-certs\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.586277 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.586197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c354f230-e34f-4e7e-a531-73798dff1995-openshift-service-ca\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.586314 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.586294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.586367 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.586326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptgwb\" (UniqueName: \"kubernetes.io/projected/c354f230-e34f-4e7e-a531-73798dff1995-kube-api-access-ptgwb\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.612314 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.612285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:53:59.687709 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.687683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.687824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.687718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptgwb\" (UniqueName: \"kubernetes.io/projected/c354f230-e34f-4e7e-a531-73798dff1995-kube-api-access-ptgwb\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.687824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.687777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-metrics-certs\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.687824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.687804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c354f230-e34f-4e7e-a531-73798dff1995-openshift-service-ca\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.687943 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:53:59.687834    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-serving-cert: secret "maas-api-e2e-derive-6956efcb-serving-cert" not found
Sep 08 18:53:59.687943 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:53:59.687913    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls podName:c354f230-e34f-4e7e-a531-73798dff1995 nodeName:}" failed. No retries permitted until 2026-09-08 18:54:00.187893516 +0000 UTC m=+1200.541986829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls") pod "maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2" (UID: "c354f230-e34f-4e7e-a531-73798dff1995") : secret "maas-api-e2e-derive-6956efcb-serving-cert" not found
Sep 08 18:53:59.688440 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.688423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c354f230-e34f-4e7e-a531-73798dff1995-openshift-service-ca\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.690167 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.690143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-metrics-certs\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.698383 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.698359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptgwb\" (UniqueName: \"kubernetes.io/projected/c354f230-e34f-4e7e-a531-73798dff1995-kube-api-access-ptgwb\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:53:59.993424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:53:59.993389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:00.127428 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.127387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:00.141130 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.141103    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:54:00.143428 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.143404    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:54:00.192893 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.192868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:54:00.195085 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.195062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls\") pod \"maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:54:00.226555 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.226530    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:00.227255 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.227236    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"]
Sep 08 18:54:00.232078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.232062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:00.236324 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.236305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6956efcb\""
Sep 08 18:54:00.236476 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.236460    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6956efcb-dockercfg-vxlpp\""
Sep 08 18:54:00.241816 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.241795    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"]
Sep 08 18:54:00.259057 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.258985    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"]
Sep 08 18:54:00.262206 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.262188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:00.271333 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.271308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"]
Sep 08 18:54:00.394490 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.394456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4bf\" (UniqueName: \"kubernetes.io/projected/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-kube-api-access-2z4bf\") pod \"payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb\" (UID: \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\") " pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:00.394629 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.394499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298ae9e0-b554-4969-a308-81cb823e8d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4\" (UID: \"298ae9e0-b554-4969-a308-81cb823e8d60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:00.394629 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.394610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-plugins-config-volume\") pod \"payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb\" (UID: \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\") " pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:00.394760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.394639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7cs\" (UniqueName: \"kubernetes.io/projected/298ae9e0-b554-4969-a308-81cb823e8d60-kube-api-access-gs7cs\") pod \"payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4\" (UID: \"298ae9e0-b554-4969-a308-81cb823e8d60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:00.425856 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.425831    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 18:54:00.435528 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.435497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:54:00.496778 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.496210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-plugins-config-volume\") pod \"payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb\" (UID: \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\") " pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:00.496778 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.496268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7cs\" (UniqueName: \"kubernetes.io/projected/298ae9e0-b554-4969-a308-81cb823e8d60-kube-api-access-gs7cs\") pod \"payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4\" (UID: \"298ae9e0-b554-4969-a308-81cb823e8d60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:00.496778 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.496354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4bf\" (UniqueName: \"kubernetes.io/projected/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-kube-api-access-2z4bf\") pod \"payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb\" (UID: \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\") " pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:00.496778 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.496395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298ae9e0-b554-4969-a308-81cb823e8d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4\" (UID: \"298ae9e0-b554-4969-a308-81cb823e8d60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:00.498266 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.498058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-plugins-config-volume\") pod \"payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb\" (UID: \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\") " pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:00.498727 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.498686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298ae9e0-b554-4969-a308-81cb823e8d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4\" (UID: \"298ae9e0-b554-4969-a308-81cb823e8d60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:00.506629 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.506599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7cs\" (UniqueName: \"kubernetes.io/projected/298ae9e0-b554-4969-a308-81cb823e8d60-kube-api-access-gs7cs\") pod \"payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4\" (UID: \"298ae9e0-b554-4969-a308-81cb823e8d60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:00.506790 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.506766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4bf\" (UniqueName: \"kubernetes.io/projected/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-kube-api-access-2z4bf\") pod \"payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb\" (UID: \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\") " pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:00.542942 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.542870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:00.573178 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.573151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:00.709449 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.709428    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"]
Sep 08 18:54:00.712818 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:54:00.712789    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc354f230_e34f_4e7e_a531_73798dff1995.slice/crio-6a1dd18230e2350166785a9647c1b8a117522b5aa028e48b78d40fc397a8f7be WatchSource:0}: Error finding container 6a1dd18230e2350166785a9647c1b8a117522b5aa028e48b78d40fc397a8f7be: Status 404 returned error can't find the container with id 6a1dd18230e2350166785a9647c1b8a117522b5aa028e48b78d40fc397a8f7be
Sep 08 18:54:00.817705 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.817674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"]
Sep 08 18:54:00.818149 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:54:00.818122    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298ae9e0_b554_4969_a308_81cb823e8d60.slice/crio-58a17ddcde940b09679dff06062390ff014014567bcce83bd531f12cd65b38a3 WatchSource:0}: Error finding container 58a17ddcde940b09679dff06062390ff014014567bcce83bd531f12cd65b38a3: Status 404 returned error can't find the container with id 58a17ddcde940b09679dff06062390ff014014567bcce83bd531f12cd65b38a3
Sep 08 18:54:00.861129 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:00.860990    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"]
Sep 08 18:54:00.863774 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:54:00.863752    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a81aed2_5565_4c80_ade4_8a6da50dfe3d.slice/crio-ae5e63a707a015a38060d5dbf9341ae93f91ef3869eb387626470e9caedc1c3a WatchSource:0}: Error finding container ae5e63a707a015a38060d5dbf9341ae93f91ef3869eb387626470e9caedc1c3a: Status 404 returned error can't find the container with id ae5e63a707a015a38060d5dbf9341ae93f91ef3869eb387626470e9caedc1c3a
Sep 08 18:54:01.071060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.070962    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb" event={"ID":"1a81aed2-5565-4c80-ade4-8a6da50dfe3d","Type":"ContainerStarted","Data":"1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450"}
Sep 08 18:54:01.071060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.071022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb" event={"ID":"1a81aed2-5565-4c80-ade4-8a6da50dfe3d","Type":"ContainerStarted","Data":"ae5e63a707a015a38060d5dbf9341ae93f91ef3869eb387626470e9caedc1c3a"}
Sep 08 18:54:01.071060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.071050    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:01.072791 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.072763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4" event={"ID":"298ae9e0-b554-4969-a308-81cb823e8d60","Type":"ContainerStarted","Data":"d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70"}
Sep 08 18:54:01.072920 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.072807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4" event={"ID":"298ae9e0-b554-4969-a308-81cb823e8d60","Type":"ContainerStarted","Data":"58a17ddcde940b09679dff06062390ff014014567bcce83bd531f12cd65b38a3"}
Sep 08 18:54:01.072920 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.072854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:01.074105 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.074084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2" event={"ID":"c354f230-e34f-4e7e-a531-73798dff1995","Type":"ContainerStarted","Data":"6a1dd18230e2350166785a9647c1b8a117522b5aa028e48b78d40fc397a8f7be"}
Sep 08 18:54:01.090709 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.090662    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb" podStartSLOduration=1.090643651 podStartE2EDuration="1.090643651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:01.089110128 +0000 UTC m=+1201.443203464" watchObservedRunningTime="2026-09-08 18:54:01.090643651 +0000 UTC m=+1201.444736987"
Sep 08 18:54:01.107805 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:01.107758    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4" podStartSLOduration=1.107741903 podStartE2EDuration="1.107741903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:01.105401919 +0000 UTC m=+1201.459495252" watchObservedRunningTime="2026-09-08 18:54:01.107741903 +0000 UTC m=+1201.461835236"
Sep 08 18:54:02.081677 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:02.081626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2" event={"ID":"c354f230-e34f-4e7e-a531-73798dff1995","Type":"ContainerStarted","Data":"e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505"}
Sep 08 18:54:02.082211 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:02.081794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:54:02.098940 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:02.098883    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2" podStartSLOduration=2.689054006 podStartE2EDuration="3.098861234s" totalImagesPullingTime="409.807228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:02.096785931 +0000 UTC m=+1202.450879263" watchObservedRunningTime="2026-09-08 18:54:02.098861234 +0000 UTC m=+1202.452954572"
Sep 08 18:54:08.092135 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:08.092105    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:54:11.172231 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.172197    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"]
Sep 08 18:54:11.175518 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.175500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.178477 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.178452    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-52ba8b\"/\"default-dockercfg-84wxf\""
Sep 08 18:54:11.178624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.178456    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-52ba8b\"/\"kube-root-ca.crt\""
Sep 08 18:54:11.178624 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.178538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-52ba8b\"/\"test-model-52ba8b-kserve-self-signed-certs\""
Sep 08 18:54:11.178967 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.178953    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-52ba8b\"/\"openshift-service-ca.crt\""
Sep 08 18:54:11.186471 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.186450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"]
Sep 08 18:54:11.201578 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.201553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-dshm\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.201696 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.201678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-home\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.201748 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.201715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce7893f-9c93-47a6-8225-ed9f17170702-tls-certs\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.201804 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.201764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-model-cache\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.201804 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.201795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hnz\" (UniqueName: \"kubernetes.io/projected/8ce7893f-9c93-47a6-8225-ed9f17170702-kube-api-access-t9hnz\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.201885 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.201821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-tmp-dir\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.302979 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.302942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-home\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.303153 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.303002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce7893f-9c93-47a6-8225-ed9f17170702-tls-certs\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.303153 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.303061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-model-cache\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.303153 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.303090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hnz\" (UniqueName: \"kubernetes.io/projected/8ce7893f-9c93-47a6-8225-ed9f17170702-kube-api-access-t9hnz\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.303153 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.303115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-tmp-dir\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.303421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.303175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-dshm\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.303492 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.303426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-home\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.303492 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.303478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-model-cache\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.303581 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.303510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-tmp-dir\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.305271 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.305249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-dshm\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.305693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.305676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce7893f-9c93-47a6-8225-ed9f17170702-tls-certs\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.311568 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.311544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hnz\" (UniqueName: \"kubernetes.io/projected/8ce7893f-9c93-47a6-8225-ed9f17170702-kube-api-access-t9hnz\") pod \"test-model-52ba8b-kserve-5c65f49d9-n27rp\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") " pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.486272 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.486189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:11.764737 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.764708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"]
Sep 08 18:54:11.766897 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:54:11.766869    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ce7893f_9c93_47a6_8225_ed9f17170702.slice/crio-1389c983efa9f0c909268657bb596eb631d7a1d2c44567bec46ace2fb380733f WatchSource:0}: Error finding container 1389c983efa9f0c909268657bb596eb631d7a1d2c44567bec46ace2fb380733f: Status 404 returned error can't find the container with id 1389c983efa9f0c909268657bb596eb631d7a1d2c44567bec46ace2fb380733f
Sep 08 18:54:11.777870 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:11.777844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:12.084184 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:12.084153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:12.084392 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:12.084222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:12.136836 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:12.136802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp" event={"ID":"8ce7893f-9c93-47a6-8225-ed9f17170702","Type":"ContainerStarted","Data":"7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1"}
Sep 08 18:54:12.137033 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:12.136851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp" event={"ID":"8ce7893f-9c93-47a6-8225-ed9f17170702","Type":"ContainerStarted","Data":"1389c983efa9f0c909268657bb596eb631d7a1d2c44567bec46ace2fb380733f"}
Sep 08 18:54:12.157494 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:12.157438    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp" podStartSLOduration=1.15742076 podStartE2EDuration="1.15742076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:12.156748153 +0000 UTC m=+1212.510841487" watchObservedRunningTime="2026-09-08 18:54:12.15742076 +0000 UTC m=+1212.511514097"
Sep 08 18:54:14.463906 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:14.463864    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"]
Sep 08 18:54:14.489398 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:14.489369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:15.460144 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:15.460101    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"]
Sep 08 18:54:15.486046 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:15.486017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:17.102798 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:17.102767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:54:17.312658 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:17.312623    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"]
Sep 08 18:54:17.340002 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:17.339964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:17.625031 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:17.624994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:20.719456 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:20.719412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:21.486567 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:21.486523    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:21.486567 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:21.486572    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:21.499379 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:21.499328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:22.198123 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:22.198089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:54:27.608656 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:27.608617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:28.300123 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.300090    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"]
Sep 08 18:54:28.300405 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.300336    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5" podUID="a2f336db-f177-4aa6-bb1f-4262186b8b88" containerName="payload-processing" containerID="cri-o://b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873" gracePeriod=30
Sep 08 18:54:28.461868 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.461833    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5" podUID="a2f336db-f177-4aa6-bb1f-4262186b8b88" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.34:9004: connect: connection refused"
Sep 08 18:54:28.557824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.557768    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:54:28.669230 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.669195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsmvh\" (UniqueName: \"kubernetes.io/projected/a2f336db-f177-4aa6-bb1f-4262186b8b88-kube-api-access-tsmvh\") pod \"a2f336db-f177-4aa6-bb1f-4262186b8b88\" (UID: \"a2f336db-f177-4aa6-bb1f-4262186b8b88\") "
Sep 08 18:54:28.669725 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.669275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f336db-f177-4aa6-bb1f-4262186b8b88-plugins-config-volume\") pod \"a2f336db-f177-4aa6-bb1f-4262186b8b88\" (UID: \"a2f336db-f177-4aa6-bb1f-4262186b8b88\") "
Sep 08 18:54:28.669725 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.669642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f336db-f177-4aa6-bb1f-4262186b8b88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2f336db-f177-4aa6-bb1f-4262186b8b88" (UID: "a2f336db-f177-4aa6-bb1f-4262186b8b88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:54:28.671383 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.671335    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f336db-f177-4aa6-bb1f-4262186b8b88-kube-api-access-tsmvh" (OuterVolumeSpecName: "kube-api-access-tsmvh") pod "a2f336db-f177-4aa6-bb1f-4262186b8b88" (UID: "a2f336db-f177-4aa6-bb1f-4262186b8b88"). InnerVolumeSpecName "kube-api-access-tsmvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:54:28.770702 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.770661    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f336db-f177-4aa6-bb1f-4262186b8b88-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:28.770702 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:28.770695    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsmvh\" (UniqueName: \"kubernetes.io/projected/a2f336db-f177-4aa6-bb1f-4262186b8b88-kube-api-access-tsmvh\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:29.115004 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.114973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:29.223546 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.223511    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2f336db-f177-4aa6-bb1f-4262186b8b88" containerID="b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873" exitCode=0
Sep 08 18:54:29.223709 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.223556    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"
Sep 08 18:54:29.223709 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.223596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5" event={"ID":"a2f336db-f177-4aa6-bb1f-4262186b8b88","Type":"ContainerDied","Data":"b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873"}
Sep 08 18:54:29.223709 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.223651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5" event={"ID":"a2f336db-f177-4aa6-bb1f-4262186b8b88","Type":"ContainerDied","Data":"da241dd8c645ff3b44f99dad10f99994f61818920582158aae1ddd9f12706b57"}
Sep 08 18:54:29.223709 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.223680    2579 scope.go:117] "RemoveContainer" containerID="b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873"
Sep 08 18:54:29.232877 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.232859    2579 scope.go:117] "RemoveContainer" containerID="b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873"
Sep 08 18:54:29.233154 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:54:29.233134    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873\": container with ID starting with b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873 not found: ID does not exist" containerID="b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873"
Sep 08 18:54:29.233223 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.233161    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873"} err="failed to get container status \"b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873\": rpc error: code = NotFound desc = could not find container \"b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873\": container with ID starting with b9b1ae59df38bd7f342f66325f766903c2f495b7ff7738ebdfd33059149cf873 not found: ID does not exist"
Sep 08 18:54:29.249621 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.249597    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"]
Sep 08 18:54:29.254695 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.254674    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-235f65-5d5f4c9bb7-wjzn5"]
Sep 08 18:54:29.483766 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.483666    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp" podUID="fbaa7c56-81c2-4bc9-9dd5-de238eab8671" containerName="main" containerID="cri-o://d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0" gracePeriod=45
Sep 08 18:54:29.733972 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.733911    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:54:29.738489 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.738470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:29.883033 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tls-certs\") pod \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") "
Sep 08 18:54:29.883218 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-dshm\") pod \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") "
Sep 08 18:54:29.883218 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883092    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s8lh\" (UniqueName: \"kubernetes.io/projected/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-kube-api-access-9s8lh\") pod \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") "
Sep 08 18:54:29.883218 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883155    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-home\") pod \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") "
Sep 08 18:54:29.883421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883282    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tmp-dir\") pod \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") "
Sep 08 18:54:29.883421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-model-cache\") pod \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\" (UID: \"fbaa7c56-81c2-4bc9-9dd5-de238eab8671\") "
Sep 08 18:54:29.883421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883398    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-home" (OuterVolumeSpecName: "home") pod "fbaa7c56-81c2-4bc9-9dd5-de238eab8671" (UID: "fbaa7c56-81c2-4bc9-9dd5-de238eab8671"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:29.883596 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbaa7c56-81c2-4bc9-9dd5-de238eab8671" (UID: "fbaa7c56-81c2-4bc9-9dd5-de238eab8671"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:29.883660 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883606    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbaa7c56-81c2-4bc9-9dd5-de238eab8671" (UID: "fbaa7c56-81c2-4bc9-9dd5-de238eab8671"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:29.883780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883760    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:29.883780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883781    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:29.883912 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.883789    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:29.885187 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.885158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbaa7c56-81c2-4bc9-9dd5-de238eab8671" (UID: "fbaa7c56-81c2-4bc9-9dd5-de238eab8671"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:54:29.885187 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.885183    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-dshm" (OuterVolumeSpecName: "dshm") pod "fbaa7c56-81c2-4bc9-9dd5-de238eab8671" (UID: "fbaa7c56-81c2-4bc9-9dd5-de238eab8671"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:29.885429 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.885242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-kube-api-access-9s8lh" (OuterVolumeSpecName: "kube-api-access-9s8lh") pod "fbaa7c56-81c2-4bc9-9dd5-de238eab8671" (UID: "fbaa7c56-81c2-4bc9-9dd5-de238eab8671"). InnerVolumeSpecName "kube-api-access-9s8lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:54:29.984436 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.984358    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:29.984436 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.984383    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:29.984436 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:29.984392    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s8lh\" (UniqueName: \"kubernetes.io/projected/fbaa7c56-81c2-4bc9-9dd5-de238eab8671-kube-api-access-9s8lh\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.164121 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.164086    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f336db-f177-4aa6-bb1f-4262186b8b88" path="/var/lib/kubelet/pods/a2f336db-f177-4aa6-bb1f-4262186b8b88/volumes"
Sep 08 18:54:30.230086 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.230057    2579 generic.go:356] "Generic (PLEG): container finished" podID="fbaa7c56-81c2-4bc9-9dd5-de238eab8671" containerID="d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0" exitCode=0
Sep 08 18:54:30.230261 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.230102    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"
Sep 08 18:54:30.230261 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.230123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp" event={"ID":"fbaa7c56-81c2-4bc9-9dd5-de238eab8671","Type":"ContainerDied","Data":"d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0"}
Sep 08 18:54:30.230261 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.230154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp" event={"ID":"fbaa7c56-81c2-4bc9-9dd5-de238eab8671","Type":"ContainerDied","Data":"7729ca5a1e7e0e134407d356929484fc6c7c65c6653afa2a5fe055209e09ba2f"}
Sep 08 18:54:30.230261 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.230176    2579 scope.go:117] "RemoveContainer" containerID="d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0"
Sep 08 18:54:30.245431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.245411    2579 scope.go:117] "RemoveContainer" containerID="d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0"
Sep 08 18:54:30.245705 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:54:30.245685    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0\": container with ID starting with d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0 not found: ID does not exist" containerID="d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0"
Sep 08 18:54:30.245751 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.245712    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0"} err="failed to get container status \"d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0\": rpc error: code = NotFound desc = could not find container \"d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0\": container with ID starting with d0c6d98e75a7641167acb06f0627e6b5406411a1d594d3fb6dcf53cd5b2d2cd0 not found: ID does not exist"
Sep 08 18:54:30.249017 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.248993    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"]
Sep 08 18:54:30.255864 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.255842    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-3edf322b-kserve-69cd8696c6-qj8sp"]
Sep 08 18:54:30.480380 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.480309    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6" podUID="f473666b-b923-4d9e-93a7-e8fe7a58636c" containerName="main" containerID="cri-o://d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70" gracePeriod=45
Sep 08 18:54:30.514536 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.514506    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"]
Sep 08 18:54:30.514767 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.514743    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2" podUID="c354f230-e34f-4e7e-a531-73798dff1995" containerName="maas-api" containerID="cri-o://e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505" gracePeriod=30
Sep 08 18:54:30.548843 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.548818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:30.604259 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.604231    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"]
Sep 08 18:54:30.604472 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.604446    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb" podUID="1a81aed2-5565-4c80-ade4-8a6da50dfe3d" containerName="payload-processing" containerID="cri-o://1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450" gracePeriod=30
Sep 08 18:54:30.624911 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.624877    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"]
Sep 08 18:54:30.625389 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.625113    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4" podUID="298ae9e0-b554-4969-a308-81cb823e8d60" containerName="payload-pre-processing" containerID="cri-o://d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70" gracePeriod=30
Sep 08 18:54:30.684386 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.684360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:30.765289 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.765267    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:54:30.805792 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.805762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:30.845021 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.844986    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:54:30.893526 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.893494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-tmp-dir\") pod \"f473666b-b923-4d9e-93a7-e8fe7a58636c\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") "
Sep 08 18:54:30.893687 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.893560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-model-cache\") pod \"f473666b-b923-4d9e-93a7-e8fe7a58636c\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") "
Sep 08 18:54:30.893687 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.893620    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/f473666b-b923-4d9e-93a7-e8fe7a58636c-kube-api-access-m2gzd\") pod \"f473666b-b923-4d9e-93a7-e8fe7a58636c\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") "
Sep 08 18:54:30.893687 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.893671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-home\") pod \"f473666b-b923-4d9e-93a7-e8fe7a58636c\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") "
Sep 08 18:54:30.893844 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.893716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-dshm\") pod \"f473666b-b923-4d9e-93a7-e8fe7a58636c\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") "
Sep 08 18:54:30.893844 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.893757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f473666b-b923-4d9e-93a7-e8fe7a58636c-tls-certs\") pod \"f473666b-b923-4d9e-93a7-e8fe7a58636c\" (UID: \"f473666b-b923-4d9e-93a7-e8fe7a58636c\") "
Sep 08 18:54:30.893844 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.893799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f473666b-b923-4d9e-93a7-e8fe7a58636c" (UID: "f473666b-b923-4d9e-93a7-e8fe7a58636c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:30.894088 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.894069    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.894375 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.894331    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-home" (OuterVolumeSpecName: "home") pod "f473666b-b923-4d9e-93a7-e8fe7a58636c" (UID: "f473666b-b923-4d9e-93a7-e8fe7a58636c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:30.894483 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.894428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f473666b-b923-4d9e-93a7-e8fe7a58636c" (UID: "f473666b-b923-4d9e-93a7-e8fe7a58636c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:30.897855 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.897825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-dshm" (OuterVolumeSpecName: "dshm") pod "f473666b-b923-4d9e-93a7-e8fe7a58636c" (UID: "f473666b-b923-4d9e-93a7-e8fe7a58636c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:30.899900 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.899875    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f473666b-b923-4d9e-93a7-e8fe7a58636c-kube-api-access-m2gzd" (OuterVolumeSpecName: "kube-api-access-m2gzd") pod "f473666b-b923-4d9e-93a7-e8fe7a58636c" (UID: "f473666b-b923-4d9e-93a7-e8fe7a58636c"). InnerVolumeSpecName "kube-api-access-m2gzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:54:30.899900 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.899890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f473666b-b923-4d9e-93a7-e8fe7a58636c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f473666b-b923-4d9e-93a7-e8fe7a58636c" (UID: "f473666b-b923-4d9e-93a7-e8fe7a58636c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:54:30.909630 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.909583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:30.919996 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.919978    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:30.922847 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.922830    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:30.994364 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994318    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptgwb\" (UniqueName: \"kubernetes.io/projected/c354f230-e34f-4e7e-a531-73798dff1995-kube-api-access-ptgwb\") pod \"c354f230-e34f-4e7e-a531-73798dff1995\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") "
Sep 08 18:54:30.994513 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-metrics-certs\") pod \"c354f230-e34f-4e7e-a531-73798dff1995\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") "
Sep 08 18:54:30.994561 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994531    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c354f230-e34f-4e7e-a531-73798dff1995-openshift-service-ca\") pod \"c354f230-e34f-4e7e-a531-73798dff1995\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") "
Sep 08 18:54:30.994601 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls\") pod \"c354f230-e34f-4e7e-a531-73798dff1995\" (UID: \"c354f230-e34f-4e7e-a531-73798dff1995\") "
Sep 08 18:54:30.994601 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994593    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298ae9e0-b554-4969-a308-81cb823e8d60-plugins-config-volume\") pod \"298ae9e0-b554-4969-a308-81cb823e8d60\" (UID: \"298ae9e0-b554-4969-a308-81cb823e8d60\") "
Sep 08 18:54:30.994780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-plugins-config-volume\") pod \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\" (UID: \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\") "
Sep 08 18:54:30.994869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994795    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs7cs\" (UniqueName: \"kubernetes.io/projected/298ae9e0-b554-4969-a308-81cb823e8d60-kube-api-access-gs7cs\") pod \"298ae9e0-b554-4969-a308-81cb823e8d60\" (UID: \"298ae9e0-b554-4969-a308-81cb823e8d60\") "
Sep 08 18:54:30.994929 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z4bf\" (UniqueName: \"kubernetes.io/projected/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-kube-api-access-2z4bf\") pod \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\" (UID: \"1a81aed2-5565-4c80-ade4-8a6da50dfe3d\") "
Sep 08 18:54:30.994929 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c354f230-e34f-4e7e-a531-73798dff1995-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c354f230-e34f-4e7e-a531-73798dff1995" (UID: "c354f230-e34f-4e7e-a531-73798dff1995"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:54:30.995029 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.994976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298ae9e0-b554-4969-a308-81cb823e8d60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "298ae9e0-b554-4969-a308-81cb823e8d60" (UID: "298ae9e0-b554-4969-a308-81cb823e8d60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:54:30.995462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.995427    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a81aed2-5565-4c80-ade4-8a6da50dfe3d" (UID: "1a81aed2-5565-4c80-ade4-8a6da50dfe3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:54:30.995462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.995438    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.995647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.995473    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/f473666b-b923-4d9e-93a7-e8fe7a58636c-kube-api-access-m2gzd\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.995647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.995489    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.995647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.995504    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c354f230-e34f-4e7e-a531-73798dff1995-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.995647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.995517    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f473666b-b923-4d9e-93a7-e8fe7a58636c-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.995647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.995531    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298ae9e0-b554-4969-a308-81cb823e8d60-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.995647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.995544    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f473666b-b923-4d9e-93a7-e8fe7a58636c-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:30.996656 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.996626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c354f230-e34f-4e7e-a531-73798dff1995" (UID: "c354f230-e34f-4e7e-a531-73798dff1995"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:54:30.996770 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.996724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c354f230-e34f-4e7e-a531-73798dff1995-kube-api-access-ptgwb" (OuterVolumeSpecName: "kube-api-access-ptgwb") pod "c354f230-e34f-4e7e-a531-73798dff1995" (UID: "c354f230-e34f-4e7e-a531-73798dff1995"). InnerVolumeSpecName "kube-api-access-ptgwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:54:30.996857 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.996830    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-kube-api-access-2z4bf" (OuterVolumeSpecName: "kube-api-access-2z4bf") pod "1a81aed2-5565-4c80-ade4-8a6da50dfe3d" (UID: "1a81aed2-5565-4c80-ade4-8a6da50dfe3d"). InnerVolumeSpecName "kube-api-access-2z4bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:54:30.996904 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.996882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c354f230-e34f-4e7e-a531-73798dff1995" (UID: "c354f230-e34f-4e7e-a531-73798dff1995"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:54:30.997156 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:30.997131    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298ae9e0-b554-4969-a308-81cb823e8d60-kube-api-access-gs7cs" (OuterVolumeSpecName: "kube-api-access-gs7cs") pod "298ae9e0-b554-4969-a308-81cb823e8d60" (UID: "298ae9e0-b554-4969-a308-81cb823e8d60"). InnerVolumeSpecName "kube-api-access-gs7cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:54:31.010809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.010759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:31.096217 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.096188    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptgwb\" (UniqueName: \"kubernetes.io/projected/c354f230-e34f-4e7e-a531-73798dff1995-kube-api-access-ptgwb\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:31.096217 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.096213    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:31.096217 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.096223    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c354f230-e34f-4e7e-a531-73798dff1995-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:31.096442 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.096231    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:31.096442 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.096239    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7cs\" (UniqueName: \"kubernetes.io/projected/298ae9e0-b554-4969-a308-81cb823e8d60-kube-api-access-gs7cs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:31.096442 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.096247    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z4bf\" (UniqueName: \"kubernetes.io/projected/1a81aed2-5565-4c80-ade4-8a6da50dfe3d-kube-api-access-2z4bf\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:31.117008 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.116981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:31.238595 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.238559    2579 generic.go:356] "Generic (PLEG): container finished" podID="f473666b-b923-4d9e-93a7-e8fe7a58636c" containerID="d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70" exitCode=0
Sep 08 18:54:31.238774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.238606    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"
Sep 08 18:54:31.238774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.238643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6" event={"ID":"f473666b-b923-4d9e-93a7-e8fe7a58636c","Type":"ContainerDied","Data":"d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70"}
Sep 08 18:54:31.238774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.238680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6" event={"ID":"f473666b-b923-4d9e-93a7-e8fe7a58636c","Type":"ContainerDied","Data":"a0cb4872cd49c9e52240c42b3c0ef0b819401472509e4cc54d21733cc0440888"}
Sep 08 18:54:31.238774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.238706    2579 scope.go:117] "RemoveContainer" containerID="d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70"
Sep 08 18:54:31.239947 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.239923    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a81aed2-5565-4c80-ade4-8a6da50dfe3d" containerID="1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450" exitCode=0
Sep 08 18:54:31.240078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.239965    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"
Sep 08 18:54:31.240078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.240011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb" event={"ID":"1a81aed2-5565-4c80-ade4-8a6da50dfe3d","Type":"ContainerDied","Data":"1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450"}
Sep 08 18:54:31.240078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.240061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb" event={"ID":"1a81aed2-5565-4c80-ade4-8a6da50dfe3d","Type":"ContainerDied","Data":"ae5e63a707a015a38060d5dbf9341ae93f91ef3869eb387626470e9caedc1c3a"}
Sep 08 18:54:31.241390 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.241369    2579 generic.go:356] "Generic (PLEG): container finished" podID="298ae9e0-b554-4969-a308-81cb823e8d60" containerID="d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70" exitCode=0
Sep 08 18:54:31.241509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.241411    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"
Sep 08 18:54:31.241509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.241424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4" event={"ID":"298ae9e0-b554-4969-a308-81cb823e8d60","Type":"ContainerDied","Data":"d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70"}
Sep 08 18:54:31.241509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.241466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4" event={"ID":"298ae9e0-b554-4969-a308-81cb823e8d60","Type":"ContainerDied","Data":"58a17ddcde940b09679dff06062390ff014014567bcce83bd531f12cd65b38a3"}
Sep 08 18:54:31.243059 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.243031    2579 generic.go:356] "Generic (PLEG): container finished" podID="c354f230-e34f-4e7e-a531-73798dff1995" containerID="e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505" exitCode=0
Sep 08 18:54:31.243189 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.243077    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"
Sep 08 18:54:31.243189 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.243143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2" event={"ID":"c354f230-e34f-4e7e-a531-73798dff1995","Type":"ContainerDied","Data":"e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505"}
Sep 08 18:54:31.243189 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.243180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2" event={"ID":"c354f230-e34f-4e7e-a531-73798dff1995","Type":"ContainerDied","Data":"6a1dd18230e2350166785a9647c1b8a117522b5aa028e48b78d40fc397a8f7be"}
Sep 08 18:54:31.249998 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.249975    2579 scope.go:117] "RemoveContainer" containerID="d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70"
Sep 08 18:54:31.250267 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:54:31.250245    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70\": container with ID starting with d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70 not found: ID does not exist" containerID="d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70"
Sep 08 18:54:31.250354 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.250278    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70"} err="failed to get container status \"d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70\": rpc error: code = NotFound desc = could not find container \"d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70\": container with ID starting with d5e54c2971273045f56958298f3f9d660d7784b03987962fc925bd40c9fa7a70 not found: ID does not exist"
Sep 08 18:54:31.250354 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.250301    2579 scope.go:117] "RemoveContainer" containerID="1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450"
Sep 08 18:54:31.259657 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.259643    2579 scope.go:117] "RemoveContainer" containerID="1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450"
Sep 08 18:54:31.259912 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:54:31.259894    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450\": container with ID starting with 1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450 not found: ID does not exist" containerID="1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450"
Sep 08 18:54:31.259967 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.259918    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450"} err="failed to get container status \"1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450\": rpc error: code = NotFound desc = could not find container \"1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450\": container with ID starting with 1078ade9b6a785a29be69fed0dba51ee7f9ad80db988bd469e0db5caedebe450 not found: ID does not exist"
Sep 08 18:54:31.259967 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.259932    2579 scope.go:117] "RemoveContainer" containerID="d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70"
Sep 08 18:54:31.265215 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.265159    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"]
Sep 08 18:54:31.267776 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.267761    2579 scope.go:117] "RemoveContainer" containerID="d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70"
Sep 08 18:54:31.268035 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:54:31.268018    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70\": container with ID starting with d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70 not found: ID does not exist" containerID="d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70"
Sep 08 18:54:31.268078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.268040    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70"} err="failed to get container status \"d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70\": rpc error: code = NotFound desc = could not find container \"d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70\": container with ID starting with d2ef12b2de81d443299e8764726834b84f58ef4133fedb2660712338371dae70 not found: ID does not exist"
Sep 08 18:54:31.268078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.268054    2579 scope.go:117] "RemoveContainer" containerID="e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505"
Sep 08 18:54:31.272058 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.272036    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-3edf322b-kserve-d9d8d848b-hlgv6"]
Sep 08 18:54:31.275988 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.275971    2579 scope.go:117] "RemoveContainer" containerID="e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505"
Sep 08 18:54:31.276226 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:54:31.276207    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505\": container with ID starting with e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505 not found: ID does not exist" containerID="e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505"
Sep 08 18:54:31.276284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.276230    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505"} err="failed to get container status \"e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505\": rpc error: code = NotFound desc = could not find container \"e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505\": container with ID starting with e8cdff370ec00d6aa0cbc5a21647b3be022c5da6ec0c13413c06df7cdba6e505 not found: ID does not exist"
Sep 08 18:54:31.287413 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.287393    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"]
Sep 08 18:54:31.294780 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.294759    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6956efcb-5886645c78-4blg4"]
Sep 08 18:54:31.312045 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.312022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"]
Sep 08 18:54:31.318148 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.318128    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6956efcb-6497486db5-hkhpb"]
Sep 08 18:54:31.332747 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.332727    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"]
Sep 08 18:54:31.337064 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.337045    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6956efcb-77fcbc5f4f-dshb2"]
Sep 08 18:54:31.923184 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:31.923150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:32.059169 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.059121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:32.158462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.158426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:32.163873 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.163849    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a81aed2-5565-4c80-ade4-8a6da50dfe3d" path="/var/lib/kubelet/pods/1a81aed2-5565-4c80-ade4-8a6da50dfe3d/volumes"
Sep 08 18:54:32.164211 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.164199    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="298ae9e0-b554-4969-a308-81cb823e8d60" path="/var/lib/kubelet/pods/298ae9e0-b554-4969-a308-81cb823e8d60/volumes"
Sep 08 18:54:32.164615 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.164600    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c354f230-e34f-4e7e-a531-73798dff1995" path="/var/lib/kubelet/pods/c354f230-e34f-4e7e-a531-73798dff1995/volumes"
Sep 08 18:54:32.165258 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.165243    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f473666b-b923-4d9e-93a7-e8fe7a58636c" path="/var/lib/kubelet/pods/f473666b-b923-4d9e-93a7-e8fe7a58636c/volumes"
Sep 08 18:54:32.165623 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.165610    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbaa7c56-81c2-4bc9-9dd5-de238eab8671" path="/var/lib/kubelet/pods/fbaa7c56-81c2-4bc9-9dd5-de238eab8671/volumes"
Sep 08 18:54:32.332632 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.332601    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v" podUID="3e614ead-0f34-4df6-828c-baa444ff648f" containerName="main" containerID="cri-o://7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c" gracePeriod=45
Sep 08 18:54:32.579882 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.579853    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:54:32.710994 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.710906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-tmp-dir\") pod \"3e614ead-0f34-4df6-828c-baa444ff648f\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") "
Sep 08 18:54:32.710994 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.710973    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-home\") pod \"3e614ead-0f34-4df6-828c-baa444ff648f\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") "
Sep 08 18:54:32.711188 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zcf8\" (UniqueName: \"kubernetes.io/projected/3e614ead-0f34-4df6-828c-baa444ff648f-kube-api-access-4zcf8\") pod \"3e614ead-0f34-4df6-828c-baa444ff648f\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") "
Sep 08 18:54:32.711188 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711060    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-model-cache\") pod \"3e614ead-0f34-4df6-828c-baa444ff648f\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") "
Sep 08 18:54:32.711188 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711120    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-dshm\") pod \"3e614ead-0f34-4df6-828c-baa444ff648f\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") "
Sep 08 18:54:32.711188 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e614ead-0f34-4df6-828c-baa444ff648f-tls-certs\") pod \"3e614ead-0f34-4df6-828c-baa444ff648f\" (UID: \"3e614ead-0f34-4df6-828c-baa444ff648f\") "
Sep 08 18:54:32.711188 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e614ead-0f34-4df6-828c-baa444ff648f" (UID: "3e614ead-0f34-4df6-828c-baa444ff648f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:32.711468 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-home" (OuterVolumeSpecName: "home") pod "3e614ead-0f34-4df6-828c-baa444ff648f" (UID: "3e614ead-0f34-4df6-828c-baa444ff648f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:32.711468 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e614ead-0f34-4df6-828c-baa444ff648f" (UID: "3e614ead-0f34-4df6-828c-baa444ff648f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:32.711552 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711475    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:32.711552 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.711488    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:32.713328 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.713305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e614ead-0f34-4df6-828c-baa444ff648f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e614ead-0f34-4df6-828c-baa444ff648f" (UID: "3e614ead-0f34-4df6-828c-baa444ff648f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:54:32.713416 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.713322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e614ead-0f34-4df6-828c-baa444ff648f-kube-api-access-4zcf8" (OuterVolumeSpecName: "kube-api-access-4zcf8") pod "3e614ead-0f34-4df6-828c-baa444ff648f" (UID: "3e614ead-0f34-4df6-828c-baa444ff648f"). InnerVolumeSpecName "kube-api-access-4zcf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:54:32.713416 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.713332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-dshm" (OuterVolumeSpecName: "dshm") pod "3e614ead-0f34-4df6-828c-baa444ff648f" (UID: "3e614ead-0f34-4df6-828c-baa444ff648f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:54:32.812854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.812818    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zcf8\" (UniqueName: \"kubernetes.io/projected/3e614ead-0f34-4df6-828c-baa444ff648f-kube-api-access-4zcf8\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:32.812854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.812849    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:32.812854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.812860    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e614ead-0f34-4df6-828c-baa444ff648f-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:32.813080 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:32.812869    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e614ead-0f34-4df6-828c-baa444ff648f-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:54:33.257867 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.257834    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e614ead-0f34-4df6-828c-baa444ff648f" containerID="7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c" exitCode=0
Sep 08 18:54:33.258310 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.257888    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"
Sep 08 18:54:33.258310 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.257926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v" event={"ID":"3e614ead-0f34-4df6-828c-baa444ff648f","Type":"ContainerDied","Data":"7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c"}
Sep 08 18:54:33.258310 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.257973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v" event={"ID":"3e614ead-0f34-4df6-828c-baa444ff648f","Type":"ContainerDied","Data":"9b0e785c6492fc722dc7dba696cbd5d697b368f7d219df1930798885168dfc43"}
Sep 08 18:54:33.258310 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.257989    2579 scope.go:117] "RemoveContainer" containerID="7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c"
Sep 08 18:54:33.266748 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.266732    2579 scope.go:117] "RemoveContainer" containerID="7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c"
Sep 08 18:54:33.267028 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:54:33.267009    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c\": container with ID starting with 7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c not found: ID does not exist" containerID="7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c"
Sep 08 18:54:33.267081 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.267036    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c"} err="failed to get container status \"7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c\": rpc error: code = NotFound desc = could not find container \"7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c\": container with ID starting with 7cfce3da5a9c057effda763d38ccd6d50b2257fa02f6b3da3b4d5b95f561fc0c not found: ID does not exist"
Sep 08 18:54:33.282496 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.282469    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"]
Sep 08 18:54:33.291700 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.291677    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-235f65/autores-model-235f65-kserve-69455d8779-tdt8v"]
Sep 08 18:54:33.677712 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:33.677676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:34.162477 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:34.162437    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e614ead-0f34-4df6-828c-baa444ff648f" path="/var/lib/kubelet/pods/3e614ead-0f34-4df6-828c-baa444ff648f/volumes"
Sep 08 18:54:34.983486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:34.983452    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:54:35.406730 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:35.406690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:35.508462 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:35.508433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:35.711963 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:35.711882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:36.242870 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:36.242802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:36.577258 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:36.577174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:36.745434 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:36.745396    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:39.348558 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:39.348515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:39.449392 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:39.449337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:40.447004 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.446973    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"]
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447353    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbaa7c56-81c2-4bc9-9dd5-de238eab8671" containerName="main"
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447365    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbaa7c56-81c2-4bc9-9dd5-de238eab8671" containerName="main"
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447376    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e614ead-0f34-4df6-828c-baa444ff648f" containerName="main"
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447381    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e614ead-0f34-4df6-828c-baa444ff648f" containerName="main"
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447393    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f473666b-b923-4d9e-93a7-e8fe7a58636c" containerName="main"
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447399    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f473666b-b923-4d9e-93a7-e8fe7a58636c" containerName="main"
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447406    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c354f230-e34f-4e7e-a531-73798dff1995" containerName="maas-api"
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447414    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c354f230-e34f-4e7e-a531-73798dff1995" containerName="maas-api"
Sep 08 18:54:40.447421 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447423    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f336db-f177-4aa6-bb1f-4262186b8b88" containerName="payload-processing"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447429    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f336db-f177-4aa6-bb1f-4262186b8b88" containerName="payload-processing"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447441    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298ae9e0-b554-4969-a308-81cb823e8d60" containerName="payload-pre-processing"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447446    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="298ae9e0-b554-4969-a308-81cb823e8d60" containerName="payload-pre-processing"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447456    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a81aed2-5565-4c80-ade4-8a6da50dfe3d" containerName="payload-processing"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447461    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a81aed2-5565-4c80-ade4-8a6da50dfe3d" containerName="payload-processing"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447536    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f473666b-b923-4d9e-93a7-e8fe7a58636c" containerName="main"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447544    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="298ae9e0-b554-4969-a308-81cb823e8d60" containerName="payload-pre-processing"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447551    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a81aed2-5565-4c80-ade4-8a6da50dfe3d" containerName="payload-processing"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447557    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e614ead-0f34-4df6-828c-baa444ff648f" containerName="main"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447565    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c354f230-e34f-4e7e-a531-73798dff1995" containerName="maas-api"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447574    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbaa7c56-81c2-4bc9-9dd5-de238eab8671" containerName="main"
Sep 08 18:54:40.447694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.447585    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f336db-f177-4aa6-bb1f-4262186b8b88" containerName="payload-processing"
Sep 08 18:54:40.452141 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.452113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.454331 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.454310    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:54:40.455019 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.454995    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-72dcc3-serving-cert\""
Sep 08 18:54:40.455137 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.455019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 18:54:40.455137 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.454997    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:54:40.455268 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.455136    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:54:40.463374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.463337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"]
Sep 08 18:54:40.484545 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.484520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:40.576529 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.576492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:40.583712 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.583686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45pdg\" (UniqueName: \"kubernetes.io/projected/279fcbd0-5520-46d9-8ced-67651e78a545-kube-api-access-45pdg\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.583845 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.583726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-metrics-certs\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.583845 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.583808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279fcbd0-5520-46d9-8ced-67651e78a545-openshift-service-ca\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.583922 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.583875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-maas-api-tls\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.685114 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.685073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279fcbd0-5520-46d9-8ced-67651e78a545-openshift-service-ca\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.685313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.685158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-maas-api-tls\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.685313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.685214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45pdg\" (UniqueName: \"kubernetes.io/projected/279fcbd0-5520-46d9-8ced-67651e78a545-kube-api-access-45pdg\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.685313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.685251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-metrics-certs\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.685827 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.685804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279fcbd0-5520-46d9-8ced-67651e78a545-openshift-service-ca\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.687537 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.687510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-maas-api-tls\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.687636 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.687623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-metrics-certs\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.694082 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.694060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45pdg\" (UniqueName: \"kubernetes.io/projected/279fcbd0-5520-46d9-8ced-67651e78a545-kube-api-access-45pdg\") pod \"maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.715310 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.715242    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:40.764111 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.764083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:40.821034 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.821001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:40.930878 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:40.930822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:41.056474 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.056396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"]
Sep 08 18:54:41.060259 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:54:41.060223    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279fcbd0_5520_46d9_8ced_67651e78a545.slice/crio-99f3a8b775ea5fa4213a498b3051cd7f7e74be13839ee45407af5c8229d47795 WatchSource:0}: Error finding container 99f3a8b775ea5fa4213a498b3051cd7f7e74be13839ee45407af5c8229d47795: Status 404 returned error can't find the container with id 99f3a8b775ea5fa4213a498b3051cd7f7e74be13839ee45407af5c8229d47795
Sep 08 18:54:41.062014 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.061995    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:54:41.069476 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.069450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:41.231113 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.231079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:41.296427 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.296395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f" event={"ID":"279fcbd0-5520-46d9-8ced-67651e78a545","Type":"ContainerStarted","Data":"99f3a8b775ea5fa4213a498b3051cd7f7e74be13839ee45407af5c8229d47795"}
Sep 08 18:54:41.330334 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.330310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:41.417296 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.417266    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"]
Sep 08 18:54:41.421839 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.421821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:41.424140 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.424121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-72dcc3\""
Sep 08 18:54:41.424227 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.424122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-72dcc3-dockercfg-lt6fd\""
Sep 08 18:54:41.440513 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.440488    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"]
Sep 08 18:54:41.448610 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.448585    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"]
Sep 08 18:54:41.451811 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.451792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:41.469196 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.469171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"]
Sep 08 18:54:41.594521 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.594496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9j65\" (UniqueName: \"kubernetes.io/projected/667848c3-71d4-4e12-b97c-95159bea703c-kube-api-access-l9j65\") pod \"payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6\" (UID: \"667848c3-71d4-4e12-b97c-95159bea703c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:41.594640 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.594541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c27195-a4aa-4e91-be2f-55389cb44cbf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n\" (UID: \"23c27195-a4aa-4e91-be2f-55389cb44cbf\") " pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:41.594640 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.594633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667848c3-71d4-4e12-b97c-95159bea703c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6\" (UID: \"667848c3-71d4-4e12-b97c-95159bea703c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:41.594768 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.594722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np9ks\" (UniqueName: \"kubernetes.io/projected/23c27195-a4aa-4e91-be2f-55389cb44cbf-kube-api-access-np9ks\") pod \"payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n\" (UID: \"23c27195-a4aa-4e91-be2f-55389cb44cbf\") " pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:41.634007 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.633977    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:41.695644 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.695616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9j65\" (UniqueName: \"kubernetes.io/projected/667848c3-71d4-4e12-b97c-95159bea703c-kube-api-access-l9j65\") pod \"payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6\" (UID: \"667848c3-71d4-4e12-b97c-95159bea703c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:41.695803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.695650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c27195-a4aa-4e91-be2f-55389cb44cbf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n\" (UID: \"23c27195-a4aa-4e91-be2f-55389cb44cbf\") " pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:41.695803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.695770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667848c3-71d4-4e12-b97c-95159bea703c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6\" (UID: \"667848c3-71d4-4e12-b97c-95159bea703c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:41.695902 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.695839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np9ks\" (UniqueName: \"kubernetes.io/projected/23c27195-a4aa-4e91-be2f-55389cb44cbf-kube-api-access-np9ks\") pod \"payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n\" (UID: \"23c27195-a4aa-4e91-be2f-55389cb44cbf\") " pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:41.696521 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.696488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667848c3-71d4-4e12-b97c-95159bea703c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6\" (UID: \"667848c3-71d4-4e12-b97c-95159bea703c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:41.696521 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.696508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c27195-a4aa-4e91-be2f-55389cb44cbf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n\" (UID: \"23c27195-a4aa-4e91-be2f-55389cb44cbf\") " pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:41.703851 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.703826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np9ks\" (UniqueName: \"kubernetes.io/projected/23c27195-a4aa-4e91-be2f-55389cb44cbf-kube-api-access-np9ks\") pod \"payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n\" (UID: \"23c27195-a4aa-4e91-be2f-55389cb44cbf\") " pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:41.704885 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.704858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9j65\" (UniqueName: \"kubernetes.io/projected/667848c3-71d4-4e12-b97c-95159bea703c-kube-api-access-l9j65\") pod \"payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6\" (UID: \"667848c3-71d4-4e12-b97c-95159bea703c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:41.732388 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.732367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:41.763275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:41.763251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:42.036748 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.036664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"]
Sep 08 18:54:42.039861 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:54:42.039831    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667848c3_71d4_4e12_b97c_95159bea703c.slice/crio-7089eb180ea79fae459b094f7bc1e04b6c774ebb8a24467e269a8fab4fb70eb4 WatchSource:0}: Error finding container 7089eb180ea79fae459b094f7bc1e04b6c774ebb8a24467e269a8fab4fb70eb4: Status 404 returned error can't find the container with id 7089eb180ea79fae459b094f7bc1e04b6c774ebb8a24467e269a8fab4fb70eb4
Sep 08 18:54:42.061274 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.061249    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"]
Sep 08 18:54:42.075479 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:54:42.075452    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c27195_a4aa_4e91_be2f_55389cb44cbf.slice/crio-1b085800359e56c7be3658dd20db60f371310a5a68540ce3f92134b63f4667ec WatchSource:0}: Error finding container 1b085800359e56c7be3658dd20db60f371310a5a68540ce3f92134b63f4667ec: Status 404 returned error can't find the container with id 1b085800359e56c7be3658dd20db60f371310a5a68540ce3f92134b63f4667ec
Sep 08 18:54:42.303739 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.303638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6" event={"ID":"667848c3-71d4-4e12-b97c-95159bea703c","Type":"ContainerStarted","Data":"891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c"}
Sep 08 18:54:42.303739 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.303704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6" event={"ID":"667848c3-71d4-4e12-b97c-95159bea703c","Type":"ContainerStarted","Data":"7089eb180ea79fae459b094f7bc1e04b6c774ebb8a24467e269a8fab4fb70eb4"}
Sep 08 18:54:42.303739 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.303737    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:54:42.305363 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.305318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f" event={"ID":"279fcbd0-5520-46d9-8ced-67651e78a545","Type":"ContainerStarted","Data":"a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083"}
Sep 08 18:54:42.305478 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.305441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:42.306943 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.306915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n" event={"ID":"23c27195-a4aa-4e91-be2f-55389cb44cbf","Type":"ContainerStarted","Data":"aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a"}
Sep 08 18:54:42.307049 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.306965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n" event={"ID":"23c27195-a4aa-4e91-be2f-55389cb44cbf","Type":"ContainerStarted","Data":"1b085800359e56c7be3658dd20db60f371310a5a68540ce3f92134b63f4667ec"}
Sep 08 18:54:42.307049 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.306995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:42.330853 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.330812    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6" podStartSLOduration=1.33079846 podStartE2EDuration="1.33079846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:42.329143641 +0000 UTC m=+1242.683236974" watchObservedRunningTime="2026-09-08 18:54:42.33079846 +0000 UTC m=+1242.684891795"
Sep 08 18:54:42.386679 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.386631    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f" podStartSLOduration=1.8996380720000001 podStartE2EDuration="2.386615697s" totalImagesPullingTime="486.977625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:42.385757912 +0000 UTC m=+1242.739851248" watchObservedRunningTime="2026-09-08 18:54:42.386615697 +0000 UTC m=+1242.740709029"
Sep 08 18:54:42.387971 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:42.387930    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n" podStartSLOduration=1.387916831 podStartE2EDuration="1.387916831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:42.360356069 +0000 UTC m=+1242.714449396" watchObservedRunningTime="2026-09-08 18:54:42.387916831 +0000 UTC m=+1242.742010163"
Sep 08 18:54:48.282154 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:48.282114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:54:48.318233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:48.318204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:54:52.091156 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:52.091120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:54:53.315040 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:53.315008    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:54:53.315539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:54:53.315078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:55:03.487298 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:03.487264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:55:04.417649 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:04.417605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:05.179185 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.179151    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"]
Sep 08 18:55:05.184435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.184415    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.187491 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.187471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c3074\"/\"test-model-9c3074-kserve-self-signed-certs\""
Sep 08 18:55:05.187604 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.187491    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c3074\"/\"openshift-service-ca.crt\""
Sep 08 18:55:05.188956 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.188932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c3074\"/\"kube-root-ca.crt\""
Sep 08 18:55:05.189782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.189765    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c3074\"/\"default-dockercfg-4trr5\""
Sep 08 18:55:05.195418 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.195396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"]
Sep 08 18:55:05.290526 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.290496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:05.320314 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.320289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-tmp-dir\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.320488 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.320321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-dshm\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.320488 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.320379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqs6h\" (UniqueName: \"kubernetes.io/projected/304461d6-51d5-4029-88a4-c049f1e6a873-kube-api-access-gqs6h\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.320488 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.320402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/304461d6-51d5-4029-88a4-c049f1e6a873-tls-certs\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.320488 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.320418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-home\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.320694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.320559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-model-cache\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421405 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-tmp-dir\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-dshm\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqs6h\" (UniqueName: \"kubernetes.io/projected/304461d6-51d5-4029-88a4-c049f1e6a873-kube-api-access-gqs6h\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/304461d6-51d5-4029-88a4-c049f1e6a873-tls-certs\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-home\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421718 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-model-cache\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421820 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-tmp-dir\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421882 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-home\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.421966 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.421906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-model-cache\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.423728 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.423711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-dshm\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.424129 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.424108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/304461d6-51d5-4029-88a4-c049f1e6a873-tls-certs\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.429246 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.429200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqs6h\" (UniqueName: \"kubernetes.io/projected/304461d6-51d5-4029-88a4-c049f1e6a873-kube-api-access-gqs6h\") pod \"test-model-9c3074-kserve-87f6bb7b9-cdptj\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") " pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.493567 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.493533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:05.765136 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:05.765048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"]
Sep 08 18:55:05.768203 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:55:05.768164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod304461d6_51d5_4029_88a4_c049f1e6a873.slice/crio-0294fc65e7f0951e5e69be524dfaab2130802b6d73681fbca9ab93f2b1542ba7 WatchSource:0}: Error finding container 0294fc65e7f0951e5e69be524dfaab2130802b6d73681fbca9ab93f2b1542ba7: Status 404 returned error can't find the container with id 0294fc65e7f0951e5e69be524dfaab2130802b6d73681fbca9ab93f2b1542ba7
Sep 08 18:55:06.425412 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:06.425373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj" event={"ID":"304461d6-51d5-4029-88a4-c049f1e6a873","Type":"ContainerStarted","Data":"f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257"}
Sep 08 18:55:06.425853 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:06.425425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj" event={"ID":"304461d6-51d5-4029-88a4-c049f1e6a873","Type":"ContainerStarted","Data":"0294fc65e7f0951e5e69be524dfaab2130802b6d73681fbca9ab93f2b1542ba7"}
Sep 08 18:55:06.443559 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:06.443505    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj" podStartSLOduration=1.443486431 podStartE2EDuration="1.443486431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:55:06.442838991 +0000 UTC m=+1266.796932324" watchObservedRunningTime="2026-09-08 18:55:06.443486431 +0000 UTC m=+1266.797579765"
Sep 08 18:55:07.645608 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:07.645562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:13.090445 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.090411    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"]
Sep 08 18:55:13.094855 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.094838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.096215 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.096191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:13.096776 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.096758    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-72dcc3\"/\"openshift-service-ca.crt\""
Sep 08 18:55:13.097294 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.097278    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-72dcc3\"/\"default-dockercfg-xc7pg\""
Sep 08 18:55:13.097452 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.097434    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-72dcc3\"/\"explicit-model-72dcc3-kserve-self-signed-certs\""
Sep 08 18:55:13.097537 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.097473    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-72dcc3\"/\"kube-root-ca.crt\""
Sep 08 18:55:13.102223 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.102203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"]
Sep 08 18:55:13.195244 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.195211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-dshm\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.195448 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.195250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc997\" (UniqueName: \"kubernetes.io/projected/a83ee190-0323-4f0a-aad3-2290d309bb96-kube-api-access-rc997\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.195448 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.195272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-tmp-dir\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.195448 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.195334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83ee190-0323-4f0a-aad3-2290d309bb96-tls-certs\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.195448 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.195374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-model-cache\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.195448 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.195422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-home\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.295948 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.295913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-home\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.296150 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.296079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-dshm\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.296150 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.296118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc997\" (UniqueName: \"kubernetes.io/projected/a83ee190-0323-4f0a-aad3-2290d309bb96-kube-api-access-rc997\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.296303 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.296152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-tmp-dir\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.296303 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.296182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83ee190-0323-4f0a-aad3-2290d309bb96-tls-certs\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.296303 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.296213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-model-cache\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.296513 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.296372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-home\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.296606 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.296570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-tmp-dir\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.296796 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.296778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-model-cache\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.298642 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.298625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-dshm\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.298819 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.298801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83ee190-0323-4f0a-aad3-2290d309bb96-tls-certs\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.303849 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.303824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc997\" (UniqueName: \"kubernetes.io/projected/a83ee190-0323-4f0a-aad3-2290d309bb96-kube-api-access-rc997\") pod \"explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") " pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.405460 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.405376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:13.678332 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.678302    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"]
Sep 08 18:55:13.680405 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:55:13.680374    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda83ee190_0323_4f0a_aad3_2290d309bb96.slice/crio-0c45c4bcc31076ed6e5ef373cf182d64538ca4634f3bbc75833682d0b4165ed5 WatchSource:0}: Error finding container 0c45c4bcc31076ed6e5ef373cf182d64538ca4634f3bbc75833682d0b4165ed5: Status 404 returned error can't find the container with id 0c45c4bcc31076ed6e5ef373cf182d64538ca4634f3bbc75833682d0b4165ed5
Sep 08 18:55:13.932969 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:13.932863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:14.046077 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:14.046040    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:14.136977 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:14.136930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:14.467606 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:14.467571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b" event={"ID":"a83ee190-0323-4f0a-aad3-2290d309bb96","Type":"ContainerStarted","Data":"dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a"}
Sep 08 18:55:14.467803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:14.467619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b" event={"ID":"a83ee190-0323-4f0a-aad3-2290d309bb96","Type":"ContainerStarted","Data":"0c45c4bcc31076ed6e5ef373cf182d64538ca4634f3bbc75833682d0b4165ed5"}
Sep 08 18:55:14.485884 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:14.485817    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b" podStartSLOduration=1.485801475 podStartE2EDuration="1.485801475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:55:14.485266453 +0000 UTC m=+1274.839359787" watchObservedRunningTime="2026-09-08 18:55:14.485801475 +0000 UTC m=+1274.839894802"
Sep 08 18:55:15.493906 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:15.493866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:15.493906 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:15.493914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:15.506509 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:15.506481    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:16.489185 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:16.489157    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:55:19.782000 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:19.781961    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:55:23.099823 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:23.099789    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"]
Sep 08 18:55:23.102699 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:23.102672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:23.405672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:23.405590    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:30.028011 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:30.027974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:33.621719 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.621683    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"]
Sep 08 18:55:33.622268 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.621925    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f" podUID="279fcbd0-5520-46d9-8ced-67651e78a545" containerName="maas-api" containerID="cri-o://a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083" gracePeriod=30
Sep 08 18:55:33.649910 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.649861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:33.677295 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.677269    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"]
Sep 08 18:55:33.677516 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.677491    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n" podUID="23c27195-a4aa-4e91-be2f-55389cb44cbf" containerName="payload-processing" containerID="cri-o://aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a" gracePeriod=30
Sep 08 18:55:33.685309 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.685286    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"]
Sep 08 18:55:33.685539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.685512    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6" podUID="667848c3-71d4-4e12-b97c-95159bea703c" containerName="payload-pre-processing" containerID="cri-o://891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c" gracePeriod=30
Sep 08 18:55:33.718022 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.717987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:33.794526 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.794498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:33.863963 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.863908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:33.949332 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.949310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:55:33.956878 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.956853    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:55:33.965195 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.965126    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:55:33.986300 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np9ks\" (UniqueName: \"kubernetes.io/projected/23c27195-a4aa-4e91-be2f-55389cb44cbf-kube-api-access-np9ks\") pod \"23c27195-a4aa-4e91-be2f-55389cb44cbf\" (UID: \"23c27195-a4aa-4e91-be2f-55389cb44cbf\") "
Sep 08 18:55:33.986459 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9j65\" (UniqueName: \"kubernetes.io/projected/667848c3-71d4-4e12-b97c-95159bea703c-kube-api-access-l9j65\") pod \"667848c3-71d4-4e12-b97c-95159bea703c\" (UID: \"667848c3-71d4-4e12-b97c-95159bea703c\") "
Sep 08 18:55:33.986459 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986366    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-metrics-certs\") pod \"279fcbd0-5520-46d9-8ced-67651e78a545\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") "
Sep 08 18:55:33.986459 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667848c3-71d4-4e12-b97c-95159bea703c-plugins-config-volume\") pod \"667848c3-71d4-4e12-b97c-95159bea703c\" (UID: \"667848c3-71d4-4e12-b97c-95159bea703c\") "
Sep 08 18:55:33.986459 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279fcbd0-5520-46d9-8ced-67651e78a545-openshift-service-ca\") pod \"279fcbd0-5520-46d9-8ced-67651e78a545\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") "
Sep 08 18:55:33.986774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-maas-api-tls\") pod \"279fcbd0-5520-46d9-8ced-67651e78a545\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") "
Sep 08 18:55:33.986774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45pdg\" (UniqueName: \"kubernetes.io/projected/279fcbd0-5520-46d9-8ced-67651e78a545-kube-api-access-45pdg\") pod \"279fcbd0-5520-46d9-8ced-67651e78a545\" (UID: \"279fcbd0-5520-46d9-8ced-67651e78a545\") "
Sep 08 18:55:33.986774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c27195-a4aa-4e91-be2f-55389cb44cbf-plugins-config-volume\") pod \"23c27195-a4aa-4e91-be2f-55389cb44cbf\" (UID: \"23c27195-a4aa-4e91-be2f-55389cb44cbf\") "
Sep 08 18:55:33.986911 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.986818    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667848c3-71d4-4e12-b97c-95159bea703c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "667848c3-71d4-4e12-b97c-95159bea703c" (UID: "667848c3-71d4-4e12-b97c-95159bea703c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:55:33.987286 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.987253    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c27195-a4aa-4e91-be2f-55389cb44cbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23c27195-a4aa-4e91-be2f-55389cb44cbf" (UID: "23c27195-a4aa-4e91-be2f-55389cb44cbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:55:33.987286 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.987271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279fcbd0-5520-46d9-8ced-67651e78a545-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "279fcbd0-5520-46d9-8ced-67651e78a545" (UID: "279fcbd0-5520-46d9-8ced-67651e78a545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:55:33.987799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.987530    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667848c3-71d4-4e12-b97c-95159bea703c-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:33.987799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.987555    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279fcbd0-5520-46d9-8ced-67651e78a545-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:33.987799 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.987583    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c27195-a4aa-4e91-be2f-55389cb44cbf-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:33.989004 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.988978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667848c3-71d4-4e12-b97c-95159bea703c-kube-api-access-l9j65" (OuterVolumeSpecName: "kube-api-access-l9j65") pod "667848c3-71d4-4e12-b97c-95159bea703c" (UID: "667848c3-71d4-4e12-b97c-95159bea703c"). InnerVolumeSpecName "kube-api-access-l9j65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:55:33.989177 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.989158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "279fcbd0-5520-46d9-8ced-67651e78a545" (UID: "279fcbd0-5520-46d9-8ced-67651e78a545"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:55:33.989268 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.989182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c27195-a4aa-4e91-be2f-55389cb44cbf-kube-api-access-np9ks" (OuterVolumeSpecName: "kube-api-access-np9ks") pod "23c27195-a4aa-4e91-be2f-55389cb44cbf" (UID: "23c27195-a4aa-4e91-be2f-55389cb44cbf"). InnerVolumeSpecName "kube-api-access-np9ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:55:33.989494 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.989330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279fcbd0-5520-46d9-8ced-67651e78a545-kube-api-access-45pdg" (OuterVolumeSpecName: "kube-api-access-45pdg") pod "279fcbd0-5520-46d9-8ced-67651e78a545" (UID: "279fcbd0-5520-46d9-8ced-67651e78a545"). InnerVolumeSpecName "kube-api-access-45pdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:55:33.989730 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:33.989698    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "279fcbd0-5520-46d9-8ced-67651e78a545" (UID: "279fcbd0-5520-46d9-8ced-67651e78a545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:55:34.088824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.088781    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np9ks\" (UniqueName: \"kubernetes.io/projected/23c27195-a4aa-4e91-be2f-55389cb44cbf-kube-api-access-np9ks\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:34.088824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.088817    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9j65\" (UniqueName: \"kubernetes.io/projected/667848c3-71d4-4e12-b97c-95159bea703c-kube-api-access-l9j65\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:34.088824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.088830    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:34.089079 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.088842    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279fcbd0-5520-46d9-8ced-67651e78a545-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:34.089079 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.088853    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45pdg\" (UniqueName: \"kubernetes.io/projected/279fcbd0-5520-46d9-8ced-67651e78a545-kube-api-access-45pdg\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:34.255773 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.255735    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:34.565798 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.565711    2579 generic.go:356] "Generic (PLEG): container finished" podID="667848c3-71d4-4e12-b97c-95159bea703c" containerID="891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c" exitCode=0
Sep 08 18:55:34.565798 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.565757    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"
Sep 08 18:55:34.565995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.565797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6" event={"ID":"667848c3-71d4-4e12-b97c-95159bea703c","Type":"ContainerDied","Data":"891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c"}
Sep 08 18:55:34.565995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.565854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6" event={"ID":"667848c3-71d4-4e12-b97c-95159bea703c","Type":"ContainerDied","Data":"7089eb180ea79fae459b094f7bc1e04b6c774ebb8a24467e269a8fab4fb70eb4"}
Sep 08 18:55:34.565995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.565883    2579 scope.go:117] "RemoveContainer" containerID="891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c"
Sep 08 18:55:34.567184 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.567147    2579 generic.go:356] "Generic (PLEG): container finished" podID="279fcbd0-5520-46d9-8ced-67651e78a545" containerID="a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083" exitCode=0
Sep 08 18:55:34.567284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.567195    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"
Sep 08 18:55:34.567284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.567208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f" event={"ID":"279fcbd0-5520-46d9-8ced-67651e78a545","Type":"ContainerDied","Data":"a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083"}
Sep 08 18:55:34.567284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.567268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f" event={"ID":"279fcbd0-5520-46d9-8ced-67651e78a545","Type":"ContainerDied","Data":"99f3a8b775ea5fa4213a498b3051cd7f7e74be13839ee45407af5c8229d47795"}
Sep 08 18:55:34.568942 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.568922    2579 generic.go:356] "Generic (PLEG): container finished" podID="23c27195-a4aa-4e91-be2f-55389cb44cbf" containerID="aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a" exitCode=0
Sep 08 18:55:34.569079 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.568964    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"
Sep 08 18:55:34.569079 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.569010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n" event={"ID":"23c27195-a4aa-4e91-be2f-55389cb44cbf","Type":"ContainerDied","Data":"aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a"}
Sep 08 18:55:34.569079 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.569057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n" event={"ID":"23c27195-a4aa-4e91-be2f-55389cb44cbf","Type":"ContainerDied","Data":"1b085800359e56c7be3658dd20db60f371310a5a68540ce3f92134b63f4667ec"}
Sep 08 18:55:34.575573 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.575556    2579 scope.go:117] "RemoveContainer" containerID="891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c"
Sep 08 18:55:34.575844 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:55:34.575816    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c\": container with ID starting with 891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c not found: ID does not exist" containerID="891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c"
Sep 08 18:55:34.575961 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.575850    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c"} err="failed to get container status \"891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c\": rpc error: code = NotFound desc = could not find container \"891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c\": container with ID starting with 891cac6d79bec48c564435fe0cbc1642a868a445e8a6e070aa66c20ed23c246c not found: ID does not exist"
Sep 08 18:55:34.575961 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.575867    2579 scope.go:117] "RemoveContainer" containerID="a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083"
Sep 08 18:55:34.583638 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.583623    2579 scope.go:117] "RemoveContainer" containerID="a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083"
Sep 08 18:55:34.583897 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:55:34.583879    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083\": container with ID starting with a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083 not found: ID does not exist" containerID="a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083"
Sep 08 18:55:34.583955 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.583904    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083"} err="failed to get container status \"a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083\": rpc error: code = NotFound desc = could not find container \"a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083\": container with ID starting with a30bd0c329d6f14b20a080aed00237150953bf67c179a3e92eba6bea53427083 not found: ID does not exist"
Sep 08 18:55:34.583955 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.583919    2579 scope.go:117] "RemoveContainer" containerID="aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a"
Sep 08 18:55:34.587964 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.587936    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"]
Sep 08 18:55:34.591784 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.591765    2579 scope.go:117] "RemoveContainer" containerID="aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a"
Sep 08 18:55:34.592029 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:55:34.592009    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a\": container with ID starting with aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a not found: ID does not exist" containerID="aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a"
Sep 08 18:55:34.592093 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.592034    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a"} err="failed to get container status \"aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a\": rpc error: code = NotFound desc = could not find container \"aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a\": container with ID starting with aa85726ba61ca507265da181b5c5b2f41ad04aab169791dc5d6f8e8c31416e6a not found: ID does not exist"
Sep 08 18:55:34.592321 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.592299    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72dcc3-7b8976dc86-84lz6"]
Sep 08 18:55:34.603285 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.603258    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"]
Sep 08 18:55:34.608664 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.608635    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-72dcc3-66bb554f6-xvb2f"]
Sep 08 18:55:34.619351 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.619325    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"]
Sep 08 18:55:34.623892 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:34.623873    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72dcc3-77bb845c7-8vx5n"]
Sep 08 18:55:35.983577 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:35.983541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:36.093222 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:36.093182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:36.162971 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:36.162939    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c27195-a4aa-4e91-be2f-55389cb44cbf" path="/var/lib/kubelet/pods/23c27195-a4aa-4e91-be2f-55389cb44cbf/volumes"
Sep 08 18:55:36.163328 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:36.163314    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="279fcbd0-5520-46d9-8ced-67651e78a545" path="/var/lib/kubelet/pods/279fcbd0-5520-46d9-8ced-67651e78a545/volumes"
Sep 08 18:55:36.163674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:36.163661    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667848c3-71d4-4e12-b97c-95159bea703c" path="/var/lib/kubelet/pods/667848c3-71d4-4e12-b97c-95159bea703c/volumes"
Sep 08 18:55:36.184303 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:36.184273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:37.787615 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:37.787578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:55:38.119315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.119276    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b" podUID="a83ee190-0323-4f0a-aad3-2290d309bb96" containerName="main" containerID="cri-o://dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a" gracePeriod=45
Sep 08 18:55:38.377276 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.377215    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:38.429768 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.429736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-dshm\") pod \"a83ee190-0323-4f0a-aad3-2290d309bb96\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") "
Sep 08 18:55:38.429768 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.429771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-home\") pod \"a83ee190-0323-4f0a-aad3-2290d309bb96\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") "
Sep 08 18:55:38.429997 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.429794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83ee190-0323-4f0a-aad3-2290d309bb96-tls-certs\") pod \"a83ee190-0323-4f0a-aad3-2290d309bb96\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") "
Sep 08 18:55:38.429997 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.429858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-model-cache\") pod \"a83ee190-0323-4f0a-aad3-2290d309bb96\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") "
Sep 08 18:55:38.429997 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.429900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc997\" (UniqueName: \"kubernetes.io/projected/a83ee190-0323-4f0a-aad3-2290d309bb96-kube-api-access-rc997\") pod \"a83ee190-0323-4f0a-aad3-2290d309bb96\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") "
Sep 08 18:55:38.429997 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.429948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-tmp-dir\") pod \"a83ee190-0323-4f0a-aad3-2290d309bb96\" (UID: \"a83ee190-0323-4f0a-aad3-2290d309bb96\") "
Sep 08 18:55:38.430210 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.430039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-home" (OuterVolumeSpecName: "home") pod "a83ee190-0323-4f0a-aad3-2290d309bb96" (UID: "a83ee190-0323-4f0a-aad3-2290d309bb96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:55:38.430210 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.430117    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-model-cache" (OuterVolumeSpecName: "model-cache") pod "a83ee190-0323-4f0a-aad3-2290d309bb96" (UID: "a83ee190-0323-4f0a-aad3-2290d309bb96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:55:38.430302 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.430270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a83ee190-0323-4f0a-aad3-2290d309bb96" (UID: "a83ee190-0323-4f0a-aad3-2290d309bb96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:55:38.430385 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.430313    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:38.430385 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.430326    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:38.431981 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.431961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-dshm" (OuterVolumeSpecName: "dshm") pod "a83ee190-0323-4f0a-aad3-2290d309bb96" (UID: "a83ee190-0323-4f0a-aad3-2290d309bb96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:55:38.432079 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.432022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83ee190-0323-4f0a-aad3-2290d309bb96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a83ee190-0323-4f0a-aad3-2290d309bb96" (UID: "a83ee190-0323-4f0a-aad3-2290d309bb96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:55:38.432079 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.432064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83ee190-0323-4f0a-aad3-2290d309bb96-kube-api-access-rc997" (OuterVolumeSpecName: "kube-api-access-rc997") pod "a83ee190-0323-4f0a-aad3-2290d309bb96" (UID: "a83ee190-0323-4f0a-aad3-2290d309bb96"). InnerVolumeSpecName "kube-api-access-rc997". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:55:38.531662 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.531627    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc997\" (UniqueName: \"kubernetes.io/projected/a83ee190-0323-4f0a-aad3-2290d309bb96-kube-api-access-rc997\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:38.531662 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.531655    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:38.531662 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.531664    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83ee190-0323-4f0a-aad3-2290d309bb96-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:38.531887 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.531672    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83ee190-0323-4f0a-aad3-2290d309bb96-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:55:38.593776 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.593745    2579 generic.go:356] "Generic (PLEG): container finished" podID="a83ee190-0323-4f0a-aad3-2290d309bb96" containerID="dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a" exitCode=0
Sep 08 18:55:38.593891 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.593784    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"
Sep 08 18:55:38.593891 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.593811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b" event={"ID":"a83ee190-0323-4f0a-aad3-2290d309bb96","Type":"ContainerDied","Data":"dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a"}
Sep 08 18:55:38.593891 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.593841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b" event={"ID":"a83ee190-0323-4f0a-aad3-2290d309bb96","Type":"ContainerDied","Data":"0c45c4bcc31076ed6e5ef373cf182d64538ca4634f3bbc75833682d0b4165ed5"}
Sep 08 18:55:38.593891 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.593858    2579 scope.go:117] "RemoveContainer" containerID="dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a"
Sep 08 18:55:38.602683 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.602665    2579 scope.go:117] "RemoveContainer" containerID="dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a"
Sep 08 18:55:38.602943 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:55:38.602923    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a\": container with ID starting with dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a not found: ID does not exist" containerID="dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a"
Sep 08 18:55:38.602982 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.602954    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a"} err="failed to get container status \"dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a\": rpc error: code = NotFound desc = could not find container \"dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a\": container with ID starting with dd2ac0bfce73b22d5ce4342c8fa724f9794951a517e62339c037ebd517f0a00a not found: ID does not exist"
Sep 08 18:55:38.612965 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.612920    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"]
Sep 08 18:55:38.614651 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:38.614631    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-72dcc3/explicit-model-72dcc3-kserve-5d9c94cf9c-fgn5b"]
Sep 08 18:55:39.940466 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:39.940430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:40.040655 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:40.040621    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:40.162647 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:40.162608    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a83ee190-0323-4f0a-aad3-2290d309bb96" path="/var/lib/kubelet/pods/a83ee190-0323-4f0a-aad3-2290d309bb96/volumes"
Sep 08 18:55:40.436561 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:40.436525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:40.542005 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:40.541970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:40.952995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:40.952967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:41.288151 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:41.288078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:41.508893 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:41.508865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:43.058994 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:43.058953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:43.158153 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:43.158120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:43.265612 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:43.265558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:46.684555 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:46.684516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:47.524829 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:47.524795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:52.168565 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:52.168533    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:55:53.345480 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:53.345447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:53.455257 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:53.455227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:53.545999 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:53.545965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:53.646317 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:53.646230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:54.380024 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:54.379988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:54.479838 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:54.479790    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:54.580610 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:54.580566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:55:56.193857 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:55:56.193826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:01.787207 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:01.787164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:01.942007 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:01.941967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:02.501469 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:02.501428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:03.666891 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.666859    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"]
Sep 08 18:56:03.667275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667207    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c27195-a4aa-4e91-be2f-55389cb44cbf" containerName="payload-processing"
Sep 08 18:56:03.667275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667218    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c27195-a4aa-4e91-be2f-55389cb44cbf" containerName="payload-processing"
Sep 08 18:56:03.667275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667230    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83ee190-0323-4f0a-aad3-2290d309bb96" containerName="main"
Sep 08 18:56:03.667275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667236    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83ee190-0323-4f0a-aad3-2290d309bb96" containerName="main"
Sep 08 18:56:03.667275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667251    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667848c3-71d4-4e12-b97c-95159bea703c" containerName="payload-pre-processing"
Sep 08 18:56:03.667275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667256    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="667848c3-71d4-4e12-b97c-95159bea703c" containerName="payload-pre-processing"
Sep 08 18:56:03.667275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667268    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="279fcbd0-5520-46d9-8ced-67651e78a545" containerName="maas-api"
Sep 08 18:56:03.667275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667273    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="279fcbd0-5520-46d9-8ced-67651e78a545" containerName="maas-api"
Sep 08 18:56:03.667643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667322    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="667848c3-71d4-4e12-b97c-95159bea703c" containerName="payload-pre-processing"
Sep 08 18:56:03.667643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667332    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83ee190-0323-4f0a-aad3-2290d309bb96" containerName="main"
Sep 08 18:56:03.667643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667354    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c27195-a4aa-4e91-be2f-55389cb44cbf" containerName="payload-processing"
Sep 08 18:56:03.667643 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.667365    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="279fcbd0-5520-46d9-8ced-67651e78a545" containerName="maas-api"
Sep 08 18:56:03.671777 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.671757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.673725 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.673685    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3b2d3e\"/\"openshift-service-ca.crt\""
Sep 08 18:56:03.674442 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.674420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3b2d3e\"/\"default-dockercfg-5v84k\""
Sep 08 18:56:03.674442 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.674435    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3b2d3e\"/\"kube-root-ca.crt\""
Sep 08 18:56:03.674602 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.674443    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3b2d3e\"/\"noten-model-3b2d3e-kserve-self-signed-certs\""
Sep 08 18:56:03.681144 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.681123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"]
Sep 08 18:56:03.691655 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.691631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:03.865241 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.865207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-tmp-dir\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.865460 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.865256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-dshm\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.865460 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.865314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a33aabc-df48-4975-893c-b9d6e073ea13-tls-certs\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.865460 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.865415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-home\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.865607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.865488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkffn\" (UniqueName: \"kubernetes.io/projected/4a33aabc-df48-4975-893c-b9d6e073ea13-kube-api-access-tkffn\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.865607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.865517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-model-cache\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.966922 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.966848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-model-cache\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.966922 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.966891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-tmp-dir\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.966922 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.966918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-dshm\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.967176 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.966938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a33aabc-df48-4975-893c-b9d6e073ea13-tls-certs\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.967176 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.966970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-home\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.967176 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.966998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkffn\" (UniqueName: \"kubernetes.io/projected/4a33aabc-df48-4975-893c-b9d6e073ea13-kube-api-access-tkffn\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.967332 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.967292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-tmp-dir\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.967332 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.967312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-model-cache\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.967449 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.967413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-home\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.969137 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.969115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-dshm\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.969488 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.969469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a33aabc-df48-4975-893c-b9d6e073ea13-tls-certs\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.974152 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.974128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkffn\" (UniqueName: \"kubernetes.io/projected/4a33aabc-df48-4975-893c-b9d6e073ea13-kube-api-access-tkffn\") pod \"noten-model-3b2d3e-kserve-5f76688db7-4k4r4\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") " pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:03.981464 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:03.981442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:04.251327 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:04.251304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"]
Sep 08 18:56:04.253351 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:56:04.253307    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a33aabc_df48_4975_893c_b9d6e073ea13.slice/crio-054b7da1aa9eae62a6b5588fd57bbd1bb0a1cc76cb735dd831e73cb81dacb31d WatchSource:0}: Error finding container 054b7da1aa9eae62a6b5588fd57bbd1bb0a1cc76cb735dd831e73cb81dacb31d: Status 404 returned error can't find the container with id 054b7da1aa9eae62a6b5588fd57bbd1bb0a1cc76cb735dd831e73cb81dacb31d
Sep 08 18:56:04.715585 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:04.715543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4" event={"ID":"4a33aabc-df48-4975-893c-b9d6e073ea13","Type":"ContainerStarted","Data":"ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a"}
Sep 08 18:56:04.716058 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:04.715595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4" event={"ID":"4a33aabc-df48-4975-893c-b9d6e073ea13","Type":"ContainerStarted","Data":"054b7da1aa9eae62a6b5588fd57bbd1bb0a1cc76cb735dd831e73cb81dacb31d"}
Sep 08 18:56:04.730478 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:04.730437    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4" podStartSLOduration=1.7304213050000001 podStartE2EDuration="1.730421305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:56:04.729815933 +0000 UTC m=+1325.083909267" watchObservedRunningTime="2026-09-08 18:56:04.730421305 +0000 UTC m=+1325.084514638"
Sep 08 18:56:06.028287 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:06.028256    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"]
Sep 08 18:56:06.041565 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:06.041533    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:06.167915 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:06.167875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:06.412002 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:06.411971    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:06.531082 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:06.531049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:08.109111 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:08.109076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:08.163199 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:08.163168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:08.352333 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:08.352300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:08.659693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:08.659658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:08.867542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:08.867505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:11.550761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:11.550729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:11.789558 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:11.789526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:11.937650 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:11.937614    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:12.892486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.892445    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"]
Sep 08 18:56:12.896118 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.896101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:12.898043 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.898022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-64effff4-serving-cert\""
Sep 08 18:56:12.898526 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.898501    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:56:12.898667 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.898552    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 18:56:12.898667 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.898556    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:56:12.898820 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.898799    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:56:12.906517 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.906494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"]
Sep 08 18:56:12.928596 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.928570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:12.947903 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.947878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-metrics-certs\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:12.948002 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.947915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl92\" (UniqueName: \"kubernetes.io/projected/83a68f3c-1629-4f4e-837d-511707220f45-kube-api-access-gwl92\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:12.948002 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.947981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:12.948086 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:12.948029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a68f3c-1629-4f4e-837d-511707220f45-openshift-service-ca\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.029217 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.029191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:13.049489 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.049461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-metrics-certs\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.049602 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.049508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl92\" (UniqueName: \"kubernetes.io/projected/83a68f3c-1629-4f4e-837d-511707220f45-kube-api-access-gwl92\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.049602 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.049551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.049602 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.049569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a68f3c-1629-4f4e-837d-511707220f45-openshift-service-ca\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.049762 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:56:13.049638    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-serving-cert: secret "maas-api-e2e-mt-64effff4-serving-cert" not found
Sep 08 18:56:13.049762 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:56:13.049712    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls podName:83a68f3c-1629-4f4e-837d-511707220f45 nodeName:}" failed. No retries permitted until 2026-09-08 18:56:13.549691154 +0000 UTC m=+1333.903784481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls") pod "maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt" (UID: "83a68f3c-1629-4f4e-837d-511707220f45") : secret "maas-api-e2e-mt-64effff4-serving-cert" not found
Sep 08 18:56:13.050271 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.050248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a68f3c-1629-4f4e-837d-511707220f45-openshift-service-ca\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.052045 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.052022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-metrics-certs\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.057165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.057142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl92\" (UniqueName: \"kubernetes.io/projected/83a68f3c-1629-4f4e-837d-511707220f45-kube-api-access-gwl92\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.372729 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.372687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:13.476013 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.475976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:13.489181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.489148    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"]
Sep 08 18:56:13.492857 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.492840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:13.494753 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.494737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-64effff4\""
Sep 08 18:56:13.503656 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.503632    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"]
Sep 08 18:56:13.531928 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.531900    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"]
Sep 08 18:56:13.534878 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.534861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:13.537103 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.537085    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-64effff4-dockercfg-z49vl\""
Sep 08 18:56:13.546574 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.546551    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"]
Sep 08 18:56:13.550030 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.549998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:13.554263 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.554242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.554367 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.554277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsfxx\" (UniqueName: \"kubernetes.io/projected/23d993a9-09ed-45ae-aea5-ab12ab32d005-kube-api-access-jsfxx\") pod \"payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd\" (UID: \"23d993a9-09ed-45ae-aea5-ab12ab32d005\") " pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:13.554367 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.554295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z\" (UID: \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:13.554464 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.554388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljggw\" (UniqueName: \"kubernetes.io/projected/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-kube-api-access-ljggw\") pod \"payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z\" (UID: \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:13.554464 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.554418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d993a9-09ed-45ae-aea5-ab12ab32d005-plugins-config-volume\") pod \"payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd\" (UID: \"23d993a9-09ed-45ae-aea5-ab12ab32d005\") " pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:13.556572 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.556556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls\") pod \"maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.655478 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.655381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsfxx\" (UniqueName: \"kubernetes.io/projected/23d993a9-09ed-45ae-aea5-ab12ab32d005-kube-api-access-jsfxx\") pod \"payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd\" (UID: \"23d993a9-09ed-45ae-aea5-ab12ab32d005\") " pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:13.655478 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.655431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z\" (UID: \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:13.655684 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.655494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljggw\" (UniqueName: \"kubernetes.io/projected/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-kube-api-access-ljggw\") pod \"payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z\" (UID: \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:13.655684 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.655518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d993a9-09ed-45ae-aea5-ab12ab32d005-plugins-config-volume\") pod \"payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd\" (UID: \"23d993a9-09ed-45ae-aea5-ab12ab32d005\") " pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:13.656174 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.656153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z\" (UID: \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:13.656242 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.656225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d993a9-09ed-45ae-aea5-ab12ab32d005-plugins-config-volume\") pod \"payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd\" (UID: \"23d993a9-09ed-45ae-aea5-ab12ab32d005\") " pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:13.663115 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.663084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljggw\" (UniqueName: \"kubernetes.io/projected/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-kube-api-access-ljggw\") pod \"payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z\" (UID: \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:13.663238 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.663138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsfxx\" (UniqueName: \"kubernetes.io/projected/23d993a9-09ed-45ae-aea5-ab12ab32d005-kube-api-access-jsfxx\") pod \"payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd\" (UID: \"23d993a9-09ed-45ae-aea5-ab12ab32d005\") " pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:13.804733 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.804685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:13.807869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.807839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:13.846012 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.845964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:13.981880 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:13.981852    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:14.083105 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.083081    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"]
Sep 08 18:56:14.085577 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:56:14.085542    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7694e2_3a3d_41f9_9123_899fc00ab6a9.slice/crio-c5e224152cec7bfa74c87ff250171ef09990f752d8c293ac122bd4e640dda82d WatchSource:0}: Error finding container c5e224152cec7bfa74c87ff250171ef09990f752d8c293ac122bd4e640dda82d: Status 404 returned error can't find the container with id c5e224152cec7bfa74c87ff250171ef09990f752d8c293ac122bd4e640dda82d
Sep 08 18:56:14.086652 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.086624    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"]
Sep 08 18:56:14.088360 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:56:14.088305    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a68f3c_1629_4f4e_837d_511707220f45.slice/crio-33cf83ceaac4331754bd639e7f388b5708436c0cf458f0d928cf2c453b851fd1 WatchSource:0}: Error finding container 33cf83ceaac4331754bd639e7f388b5708436c0cf458f0d928cf2c453b851fd1: Status 404 returned error can't find the container with id 33cf83ceaac4331754bd639e7f388b5708436c0cf458f0d928cf2c453b851fd1
Sep 08 18:56:14.131408 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.131386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"]
Sep 08 18:56:14.134224 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:56:14.134199    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d993a9_09ed_45ae_aea5_ab12ab32d005.slice/crio-4b0d2a7fd3d55e3c6daf85bada0ea7d24ff3d8b2a639412bbe4d5a91802b3029 WatchSource:0}: Error finding container 4b0d2a7fd3d55e3c6daf85bada0ea7d24ff3d8b2a639412bbe4d5a91802b3029: Status 404 returned error can't find the container with id 4b0d2a7fd3d55e3c6daf85bada0ea7d24ff3d8b2a639412bbe4d5a91802b3029
Sep 08 18:56:14.775565 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.775531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd" event={"ID":"23d993a9-09ed-45ae-aea5-ab12ab32d005","Type":"ContainerStarted","Data":"7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69"}
Sep 08 18:56:14.775758 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.775599    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:14.775758 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.775627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd" event={"ID":"23d993a9-09ed-45ae-aea5-ab12ab32d005","Type":"ContainerStarted","Data":"4b0d2a7fd3d55e3c6daf85bada0ea7d24ff3d8b2a639412bbe4d5a91802b3029"}
Sep 08 18:56:14.777377 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.777336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z" event={"ID":"4c7694e2-3a3d-41f9-9123-899fc00ab6a9","Type":"ContainerStarted","Data":"8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f"}
Sep 08 18:56:14.777511 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.777400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z" event={"ID":"4c7694e2-3a3d-41f9-9123-899fc00ab6a9","Type":"ContainerStarted","Data":"c5e224152cec7bfa74c87ff250171ef09990f752d8c293ac122bd4e640dda82d"}
Sep 08 18:56:14.777511 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.777430    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:14.778717 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.778692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt" event={"ID":"83a68f3c-1629-4f4e-837d-511707220f45","Type":"ContainerStarted","Data":"33cf83ceaac4331754bd639e7f388b5708436c0cf458f0d928cf2c453b851fd1"}
Sep 08 18:56:14.792094 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.792052    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd" podStartSLOduration=1.7920384440000001 podStartE2EDuration="1.792038444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:56:14.790256306 +0000 UTC m=+1335.144349639" watchObservedRunningTime="2026-09-08 18:56:14.792038444 +0000 UTC m=+1335.146131777"
Sep 08 18:56:14.806156 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:14.806108    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z" podStartSLOduration=1.80609011 podStartE2EDuration="1.80609011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:56:14.805008486 +0000 UTC m=+1335.159101819" watchObservedRunningTime="2026-09-08 18:56:14.80609011 +0000 UTC m=+1335.160183444"
Sep 08 18:56:15.785486 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:15.785445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt" event={"ID":"83a68f3c-1629-4f4e-837d-511707220f45","Type":"ContainerStarted","Data":"ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b"}
Sep 08 18:56:15.785953 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:15.785662    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:15.803498 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:15.803449    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt" podStartSLOduration=3.078125889 podStartE2EDuration="3.803431465s" totalImagesPullingTime="725.305576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:56:15.803066462 +0000 UTC m=+1336.157159796" watchObservedRunningTime="2026-09-08 18:56:15.803431465 +0000 UTC m=+1336.157524800"
Sep 08 18:56:18.281889 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:18.281851    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:56:18.354774 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:18.354742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"]
Sep 08 18:56:18.376824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:18.376784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:19.553809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:19.553777    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"]
Sep 08 18:56:19.572288 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:19.572261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:19.904745 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:19.904711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:21.743666 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:21.743615    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4" podUID="4a33aabc-df48-4975-893c-b9d6e073ea13" containerName="main" containerID="cri-o://ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a" gracePeriod=45
Sep 08 18:56:21.795503 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:21.795480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:21.992315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:21.992291    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:22.037431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037358    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a33aabc-df48-4975-893c-b9d6e073ea13-tls-certs\") pod \"4a33aabc-df48-4975-893c-b9d6e073ea13\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") "
Sep 08 18:56:22.037431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037400    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-tmp-dir\") pod \"4a33aabc-df48-4975-893c-b9d6e073ea13\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") "
Sep 08 18:56:22.037639 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-home\") pod \"4a33aabc-df48-4975-893c-b9d6e073ea13\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") "
Sep 08 18:56:22.037639 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037567    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-model-cache\") pod \"4a33aabc-df48-4975-893c-b9d6e073ea13\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") "
Sep 08 18:56:22.037639 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037617    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkffn\" (UniqueName: \"kubernetes.io/projected/4a33aabc-df48-4975-893c-b9d6e073ea13-kube-api-access-tkffn\") pod \"4a33aabc-df48-4975-893c-b9d6e073ea13\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") "
Sep 08 18:56:22.037786 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037662    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a33aabc-df48-4975-893c-b9d6e073ea13" (UID: "4a33aabc-df48-4975-893c-b9d6e073ea13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:22.037786 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-dshm\") pod \"4a33aabc-df48-4975-893c-b9d6e073ea13\" (UID: \"4a33aabc-df48-4975-893c-b9d6e073ea13\") "
Sep 08 18:56:22.037894 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-home" (OuterVolumeSpecName: "home") pod "4a33aabc-df48-4975-893c-b9d6e073ea13" (UID: "4a33aabc-df48-4975-893c-b9d6e073ea13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:22.037894 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.037844    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a33aabc-df48-4975-893c-b9d6e073ea13" (UID: "4a33aabc-df48-4975-893c-b9d6e073ea13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:22.038139 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.038114    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:22.038139 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.038139    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:22.038315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.038151    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:22.039839 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.039809    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a33aabc-df48-4975-893c-b9d6e073ea13-kube-api-access-tkffn" (OuterVolumeSpecName: "kube-api-access-tkffn") pod "4a33aabc-df48-4975-893c-b9d6e073ea13" (UID: "4a33aabc-df48-4975-893c-b9d6e073ea13"). InnerVolumeSpecName "kube-api-access-tkffn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:56:22.039975 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.039958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a33aabc-df48-4975-893c-b9d6e073ea13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a33aabc-df48-4975-893c-b9d6e073ea13" (UID: "4a33aabc-df48-4975-893c-b9d6e073ea13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:56:22.040031 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.039973    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-dshm" (OuterVolumeSpecName: "dshm") pod "4a33aabc-df48-4975-893c-b9d6e073ea13" (UID: "4a33aabc-df48-4975-893c-b9d6e073ea13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:22.138785 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.138748    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkffn\" (UniqueName: \"kubernetes.io/projected/4a33aabc-df48-4975-893c-b9d6e073ea13-kube-api-access-tkffn\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:22.138785 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.138779    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a33aabc-df48-4975-893c-b9d6e073ea13-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:22.138785 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.138790    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a33aabc-df48-4975-893c-b9d6e073ea13-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:22.819514 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.819481    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a33aabc-df48-4975-893c-b9d6e073ea13" containerID="ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a" exitCode=0
Sep 08 18:56:22.819946 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.819527    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"
Sep 08 18:56:22.819946 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.819559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4" event={"ID":"4a33aabc-df48-4975-893c-b9d6e073ea13","Type":"ContainerDied","Data":"ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a"}
Sep 08 18:56:22.819946 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.819598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4" event={"ID":"4a33aabc-df48-4975-893c-b9d6e073ea13","Type":"ContainerDied","Data":"054b7da1aa9eae62a6b5588fd57bbd1bb0a1cc76cb735dd831e73cb81dacb31d"}
Sep 08 18:56:22.819946 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.819618    2579 scope.go:117] "RemoveContainer" containerID="ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a"
Sep 08 18:56:22.828131 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.828115    2579 scope.go:117] "RemoveContainer" containerID="ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a"
Sep 08 18:56:22.828392 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:56:22.828372    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a\": container with ID starting with ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a not found: ID does not exist" containerID="ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a"
Sep 08 18:56:22.828468 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.828399    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a"} err="failed to get container status \"ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a\": rpc error: code = NotFound desc = could not find container \"ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a\": container with ID starting with ec7f4b17c49b3c8d79e2bc731e5840eb4705e89fe6d2eeb9620b41211b98a76a not found: ID does not exist"
Sep 08 18:56:22.834608 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.834587    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"]
Sep 08 18:56:22.837778 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:22.837759    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-3b2d3e/noten-model-3b2d3e-kserve-5f76688db7-4k4r4"]
Sep 08 18:56:24.162781 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:24.162747    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a33aabc-df48-4975-893c-b9d6e073ea13" path="/var/lib/kubelet/pods/4a33aabc-df48-4975-893c-b9d6e073ea13/volumes"
Sep 08 18:56:25.788011 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:25.787983    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:25.788413 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:25.788048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:26.176496 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:26.176459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:56:30.215739 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:30.215699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:30.320956 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:30.320916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:30.421165 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:30.421136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:30.492809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:30.492724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:30.549867 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:30.549839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:30.879435 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:30.879398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:33.374827 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.374782    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp" podUID="8ce7893f-9c93-47a6-8225-ed9f17170702" containerName="main" containerID="cri-o://7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1" gracePeriod=45
Sep 08 18:56:33.566238 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.566208    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:56:33.614877 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.614849    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:56:33.748824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.748747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9hnz\" (UniqueName: \"kubernetes.io/projected/8ce7893f-9c93-47a6-8225-ed9f17170702-kube-api-access-t9hnz\") pod \"8ce7893f-9c93-47a6-8225-ed9f17170702\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") "
Sep 08 18:56:33.748824 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.748793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce7893f-9c93-47a6-8225-ed9f17170702-tls-certs\") pod \"8ce7893f-9c93-47a6-8225-ed9f17170702\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") "
Sep 08 18:56:33.749078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.748827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-tmp-dir\") pod \"8ce7893f-9c93-47a6-8225-ed9f17170702\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") "
Sep 08 18:56:33.749078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.748848    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-home\") pod \"8ce7893f-9c93-47a6-8225-ed9f17170702\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") "
Sep 08 18:56:33.749078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.748877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-dshm\") pod \"8ce7893f-9c93-47a6-8225-ed9f17170702\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") "
Sep 08 18:56:33.749078 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.748899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-model-cache\") pod \"8ce7893f-9c93-47a6-8225-ed9f17170702\" (UID: \"8ce7893f-9c93-47a6-8225-ed9f17170702\") "
Sep 08 18:56:33.749283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.749186    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-home" (OuterVolumeSpecName: "home") pod "8ce7893f-9c93-47a6-8225-ed9f17170702" (UID: "8ce7893f-9c93-47a6-8225-ed9f17170702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:33.749283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.749191    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ce7893f-9c93-47a6-8225-ed9f17170702" (UID: "8ce7893f-9c93-47a6-8225-ed9f17170702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:33.749283 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.749259    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ce7893f-9c93-47a6-8225-ed9f17170702" (UID: "8ce7893f-9c93-47a6-8225-ed9f17170702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:33.751134 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.751110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-dshm" (OuterVolumeSpecName: "dshm") pod "8ce7893f-9c93-47a6-8225-ed9f17170702" (UID: "8ce7893f-9c93-47a6-8225-ed9f17170702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:33.751134 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.751126    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce7893f-9c93-47a6-8225-ed9f17170702-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ce7893f-9c93-47a6-8225-ed9f17170702" (UID: "8ce7893f-9c93-47a6-8225-ed9f17170702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:56:33.751262 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.751212    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce7893f-9c93-47a6-8225-ed9f17170702-kube-api-access-t9hnz" (OuterVolumeSpecName: "kube-api-access-t9hnz") pod "8ce7893f-9c93-47a6-8225-ed9f17170702" (UID: "8ce7893f-9c93-47a6-8225-ed9f17170702"). InnerVolumeSpecName "kube-api-access-t9hnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:56:33.850118 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.850082    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:33.850118 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.850110    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:33.850118 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.850121    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9hnz\" (UniqueName: \"kubernetes.io/projected/8ce7893f-9c93-47a6-8225-ed9f17170702-kube-api-access-t9hnz\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:33.850118 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.850130    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce7893f-9c93-47a6-8225-ed9f17170702-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:33.850414 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.850138    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:33.850414 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.850145    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce7893f-9c93-47a6-8225-ed9f17170702-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:33.876787 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.876745    2579 generic.go:356] "Generic (PLEG): container finished" podID="8ce7893f-9c93-47a6-8225-ed9f17170702" containerID="7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1" exitCode=0
Sep 08 18:56:33.876787 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.876788    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"
Sep 08 18:56:33.876983 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.876825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp" event={"ID":"8ce7893f-9c93-47a6-8225-ed9f17170702","Type":"ContainerDied","Data":"7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1"}
Sep 08 18:56:33.876983 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.876872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp" event={"ID":"8ce7893f-9c93-47a6-8225-ed9f17170702","Type":"ContainerDied","Data":"1389c983efa9f0c909268657bb596eb631d7a1d2c44567bec46ace2fb380733f"}
Sep 08 18:56:33.876983 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.876891    2579 scope.go:117] "RemoveContainer" containerID="7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1"
Sep 08 18:56:33.885930 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.885866    2579 scope.go:117] "RemoveContainer" containerID="7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1"
Sep 08 18:56:33.886170 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:56:33.886154    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1\": container with ID starting with 7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1 not found: ID does not exist" containerID="7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1"
Sep 08 18:56:33.886211 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.886179    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1"} err="failed to get container status \"7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1\": rpc error: code = NotFound desc = could not find container \"7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1\": container with ID starting with 7dfd55d15c65a8422a471852c5cd6e23ffb273036cd0c46054daa317601840b1 not found: ID does not exist"
Sep 08 18:56:33.900798 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.900767    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"]
Sep 08 18:56:33.903364 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:33.903324    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-52ba8b/test-model-52ba8b-kserve-5c65f49d9-n27rp"]
Sep 08 18:56:34.163415 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.163379    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce7893f-9c93-47a6-8225-ed9f17170702" path="/var/lib/kubelet/pods/8ce7893f-9c93-47a6-8225-ed9f17170702/volumes"
Sep 08 18:56:34.574057 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.573939    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj" podUID="304461d6-51d5-4029-88a4-c049f1e6a873" containerName="main" containerID="cri-o://f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257" gracePeriod=45
Sep 08 18:56:34.686627 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.686596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:34.826313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.826254    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:56:34.883049 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.883017    2579 generic.go:356] "Generic (PLEG): container finished" podID="304461d6-51d5-4029-88a4-c049f1e6a873" containerID="f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257" exitCode=0
Sep 08 18:56:34.883233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.883055    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"
Sep 08 18:56:34.883233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.883099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj" event={"ID":"304461d6-51d5-4029-88a4-c049f1e6a873","Type":"ContainerDied","Data":"f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257"}
Sep 08 18:56:34.883233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.883138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj" event={"ID":"304461d6-51d5-4029-88a4-c049f1e6a873","Type":"ContainerDied","Data":"0294fc65e7f0951e5e69be524dfaab2130802b6d73681fbca9ab93f2b1542ba7"}
Sep 08 18:56:34.883233 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.883157    2579 scope.go:117] "RemoveContainer" containerID="f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257"
Sep 08 18:56:34.891580 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.891563    2579 scope.go:117] "RemoveContainer" containerID="f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257"
Sep 08 18:56:34.891853 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:56:34.891832    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257\": container with ID starting with f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257 not found: ID does not exist" containerID="f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257"
Sep 08 18:56:34.891921 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.891858    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257"} err="failed to get container status \"f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257\": rpc error: code = NotFound desc = could not find container \"f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257\": container with ID starting with f1ed426574448853964fdc4f1a026823db0e4761adf4416ae7ba50ef69f82257 not found: ID does not exist"
Sep 08 18:56:34.961562 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961522    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqs6h\" (UniqueName: \"kubernetes.io/projected/304461d6-51d5-4029-88a4-c049f1e6a873-kube-api-access-gqs6h\") pod \"304461d6-51d5-4029-88a4-c049f1e6a873\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") "
Sep 08 18:56:34.961562 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961563    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/304461d6-51d5-4029-88a4-c049f1e6a873-tls-certs\") pod \"304461d6-51d5-4029-88a4-c049f1e6a873\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") "
Sep 08 18:56:34.961782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961616    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-dshm\") pod \"304461d6-51d5-4029-88a4-c049f1e6a873\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") "
Sep 08 18:56:34.961782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-tmp-dir\") pod \"304461d6-51d5-4029-88a4-c049f1e6a873\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") "
Sep 08 18:56:34.961782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961662    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-model-cache\") pod \"304461d6-51d5-4029-88a4-c049f1e6a873\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") "
Sep 08 18:56:34.961782 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961682    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-home\") pod \"304461d6-51d5-4029-88a4-c049f1e6a873\" (UID: \"304461d6-51d5-4029-88a4-c049f1e6a873\") "
Sep 08 18:56:34.961986 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961946    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-model-cache" (OuterVolumeSpecName: "model-cache") pod "304461d6-51d5-4029-88a4-c049f1e6a873" (UID: "304461d6-51d5-4029-88a4-c049f1e6a873"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:34.962050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961955    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "304461d6-51d5-4029-88a4-c049f1e6a873" (UID: "304461d6-51d5-4029-88a4-c049f1e6a873"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:34.962050 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.961987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-home" (OuterVolumeSpecName: "home") pod "304461d6-51d5-4029-88a4-c049f1e6a873" (UID: "304461d6-51d5-4029-88a4-c049f1e6a873"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:34.962124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.962060    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:34.962124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.962078    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:34.962124 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.962087    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:34.963881 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.963859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304461d6-51d5-4029-88a4-c049f1e6a873-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "304461d6-51d5-4029-88a4-c049f1e6a873" (UID: "304461d6-51d5-4029-88a4-c049f1e6a873"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:56:34.964152 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.964127    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-dshm" (OuterVolumeSpecName: "dshm") pod "304461d6-51d5-4029-88a4-c049f1e6a873" (UID: "304461d6-51d5-4029-88a4-c049f1e6a873"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:34.964251 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:34.964151    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304461d6-51d5-4029-88a4-c049f1e6a873-kube-api-access-gqs6h" (OuterVolumeSpecName: "kube-api-access-gqs6h") pod "304461d6-51d5-4029-88a4-c049f1e6a873" (UID: "304461d6-51d5-4029-88a4-c049f1e6a873"). InnerVolumeSpecName "kube-api-access-gqs6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:56:35.063103 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:35.063051    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/304461d6-51d5-4029-88a4-c049f1e6a873-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:35.063103 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:35.063095    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/304461d6-51d5-4029-88a4-c049f1e6a873-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:35.063103 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:35.063106    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqs6h\" (UniqueName: \"kubernetes.io/projected/304461d6-51d5-4029-88a4-c049f1e6a873-kube-api-access-gqs6h\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:35.203096 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:35.203068    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"]
Sep 08 18:56:35.207554 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:35.207527    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9c3074/test-model-9c3074-kserve-87f6bb7b9-cdptj"]
Sep 08 18:56:36.163003 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:36.162962    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="304461d6-51d5-4029-88a4-c049f1e6a873" path="/var/lib/kubelet/pods/304461d6-51d5-4029-88a4-c049f1e6a873/volumes"
Sep 08 18:56:36.852362 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:36.852311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:36.952265 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:36.952230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:37.052438 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:37.052403    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:37.261060 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:37.260985    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:37.585602 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:37.585571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:37.699735 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:37.699706    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:37.799323 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:37.799290    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:38.127406 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:38.127374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:39.259423 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:39.259388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:43.820485 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:43.820442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:43.879016 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:43.878990    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:44.336904 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:44.336875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:47.797235 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:47.797200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:48.361265 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:48.361236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:48.444007 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:48.443965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:48.499373 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:48.499324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:48.599649 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:48.599620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:48.916783 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:48.916749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:49.016676 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:49.016629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:51.276938 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:51.276904    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:56:54.353556 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:54.353522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:54.893431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:54.893401    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"]
Sep 08 18:56:54.893636 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:54.893613    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt" podUID="83a68f3c-1629-4f4e-837d-511707220f45" containerName="maas-api" containerID="cri-o://ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b" gracePeriod=30
Sep 08 18:56:54.925446 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:54.925410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:54.993936 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:54.993904    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"]
Sep 08 18:56:54.994153 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:54.994110    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd" podUID="23d993a9-09ed-45ae-aea5-ab12ab32d005" containerName="payload-processing" containerID="cri-o://7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69" gracePeriod=30
Sep 08 18:56:55.017200 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.017170    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"]
Sep 08 18:56:55.017524 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.017462    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z" podUID="4c7694e2-3a3d-41f9-9123-899fc00ab6a9" containerName="payload-pre-processing" containerID="cri-o://8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f" gracePeriod=30
Sep 08 18:56:55.049518 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.049488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:55.150787 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.150657    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:55.175937 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.175685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:55.249095 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.249069    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls\") pod \"83a68f3c-1629-4f4e-837d-511707220f45\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") "
Sep 08 18:56:55.249218 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.249137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-metrics-certs\") pod \"83a68f3c-1629-4f4e-837d-511707220f45\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") "
Sep 08 18:56:55.249218 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.249165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a68f3c-1629-4f4e-837d-511707220f45-openshift-service-ca\") pod \"83a68f3c-1629-4f4e-837d-511707220f45\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") "
Sep 08 18:56:55.249218 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.249212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwl92\" (UniqueName: \"kubernetes.io/projected/83a68f3c-1629-4f4e-837d-511707220f45-kube-api-access-gwl92\") pod \"83a68f3c-1629-4f4e-837d-511707220f45\" (UID: \"83a68f3c-1629-4f4e-837d-511707220f45\") "
Sep 08 18:56:55.249618 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.249585    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a68f3c-1629-4f4e-837d-511707220f45-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83a68f3c-1629-4f4e-837d-511707220f45" (UID: "83a68f3c-1629-4f4e-837d-511707220f45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:56:55.252690 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.252665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "83a68f3c-1629-4f4e-837d-511707220f45" (UID: "83a68f3c-1629-4f4e-837d-511707220f45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:56:55.254681 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.254652    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83a68f3c-1629-4f4e-837d-511707220f45" (UID: "83a68f3c-1629-4f4e-837d-511707220f45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:56:55.254803 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.254780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a68f3c-1629-4f4e-837d-511707220f45-kube-api-access-gwl92" (OuterVolumeSpecName: "kube-api-access-gwl92") pod "83a68f3c-1629-4f4e-837d-511707220f45" (UID: "83a68f3c-1629-4f4e-837d-511707220f45"). InnerVolumeSpecName "kube-api-access-gwl92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:56:55.267937 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.267878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:55.284534 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.284210    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:55.295214 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.295191    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:55.337930 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.337898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:55.350481 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d993a9-09ed-45ae-aea5-ab12ab32d005-plugins-config-volume\") pod \"23d993a9-09ed-45ae-aea5-ab12ab32d005\" (UID: \"23d993a9-09ed-45ae-aea5-ab12ab32d005\") "
Sep 08 18:56:55.350613 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350499    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljggw\" (UniqueName: \"kubernetes.io/projected/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-kube-api-access-ljggw\") pod \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\" (UID: \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\") "
Sep 08 18:56:55.350613 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-plugins-config-volume\") pod \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\" (UID: \"4c7694e2-3a3d-41f9-9123-899fc00ab6a9\") "
Sep 08 18:56:55.350613 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350562    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsfxx\" (UniqueName: \"kubernetes.io/projected/23d993a9-09ed-45ae-aea5-ab12ab32d005-kube-api-access-jsfxx\") pod \"23d993a9-09ed-45ae-aea5-ab12ab32d005\" (UID: \"23d993a9-09ed-45ae-aea5-ab12ab32d005\") "
Sep 08 18:56:55.350769 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350757    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:55.350809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350771    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a68f3c-1629-4f4e-837d-511707220f45-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:55.350809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350780    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a68f3c-1629-4f4e-837d-511707220f45-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:55.350809 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350790    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwl92\" (UniqueName: \"kubernetes.io/projected/83a68f3c-1629-4f4e-837d-511707220f45-kube-api-access-gwl92\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:55.350913 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d993a9-09ed-45ae-aea5-ab12ab32d005-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23d993a9-09ed-45ae-aea5-ab12ab32d005" (UID: "23d993a9-09ed-45ae-aea5-ab12ab32d005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:56:55.350953 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.350933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c7694e2-3a3d-41f9-9123-899fc00ab6a9" (UID: "4c7694e2-3a3d-41f9-9123-899fc00ab6a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:56:55.352652 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.352627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d993a9-09ed-45ae-aea5-ab12ab32d005-kube-api-access-jsfxx" (OuterVolumeSpecName: "kube-api-access-jsfxx") pod "23d993a9-09ed-45ae-aea5-ab12ab32d005" (UID: "23d993a9-09ed-45ae-aea5-ab12ab32d005"). InnerVolumeSpecName "kube-api-access-jsfxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:56:55.352741 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.352665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-kube-api-access-ljggw" (OuterVolumeSpecName: "kube-api-access-ljggw") pod "4c7694e2-3a3d-41f9-9123-899fc00ab6a9" (UID: "4c7694e2-3a3d-41f9-9123-899fc00ab6a9"). InnerVolumeSpecName "kube-api-access-ljggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:56:55.451761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.451682    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d993a9-09ed-45ae-aea5-ab12ab32d005-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:55.451761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.451708    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljggw\" (UniqueName: \"kubernetes.io/projected/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-kube-api-access-ljggw\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:55.451761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.451718    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7694e2-3a3d-41f9-9123-899fc00ab6a9-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:55.451761 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.451728    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsfxx\" (UniqueName: \"kubernetes.io/projected/23d993a9-09ed-45ae-aea5-ab12ab32d005-kube-api-access-jsfxx\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:56:55.671918 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.671883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:55.923971 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.923941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:55.984492 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.984447    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c7694e2-3a3d-41f9-9123-899fc00ab6a9" containerID="8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f" exitCode=0
Sep 08 18:56:55.984672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.984502    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"
Sep 08 18:56:55.984672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.984529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z" event={"ID":"4c7694e2-3a3d-41f9-9123-899fc00ab6a9","Type":"ContainerDied","Data":"8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f"}
Sep 08 18:56:55.984672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.984579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z" event={"ID":"4c7694e2-3a3d-41f9-9123-899fc00ab6a9","Type":"ContainerDied","Data":"c5e224152cec7bfa74c87ff250171ef09990f752d8c293ac122bd4e640dda82d"}
Sep 08 18:56:55.984672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.984601    2579 scope.go:117] "RemoveContainer" containerID="8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f"
Sep 08 18:56:55.986005 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.985985    2579 generic.go:356] "Generic (PLEG): container finished" podID="83a68f3c-1629-4f4e-837d-511707220f45" containerID="ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b" exitCode=0
Sep 08 18:56:55.986076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.986046    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"
Sep 08 18:56:55.986132 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.986071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt" event={"ID":"83a68f3c-1629-4f4e-837d-511707220f45","Type":"ContainerDied","Data":"ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b"}
Sep 08 18:56:55.986132 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.986111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt" event={"ID":"83a68f3c-1629-4f4e-837d-511707220f45","Type":"ContainerDied","Data":"33cf83ceaac4331754bd639e7f388b5708436c0cf458f0d928cf2c453b851fd1"}
Sep 08 18:56:55.987431 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.987410    2579 generic.go:356] "Generic (PLEG): container finished" podID="23d993a9-09ed-45ae-aea5-ab12ab32d005" containerID="7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69" exitCode=0
Sep 08 18:56:55.987550 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.987442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd" event={"ID":"23d993a9-09ed-45ae-aea5-ab12ab32d005","Type":"ContainerDied","Data":"7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69"}
Sep 08 18:56:55.987550 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.987461    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"
Sep 08 18:56:55.987550 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.987478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd" event={"ID":"23d993a9-09ed-45ae-aea5-ab12ab32d005","Type":"ContainerDied","Data":"4b0d2a7fd3d55e3c6daf85bada0ea7d24ff3d8b2a639412bbe4d5a91802b3029"}
Sep 08 18:56:55.994029 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.994008    2579 scope.go:117] "RemoveContainer" containerID="8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f"
Sep 08 18:56:55.994351 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:56:55.994311    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f\": container with ID starting with 8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f not found: ID does not exist" containerID="8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f"
Sep 08 18:56:55.994440 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.994366    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f"} err="failed to get container status \"8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f\": rpc error: code = NotFound desc = could not find container \"8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f\": container with ID starting with 8c9d3d0747277884aa5e7533cc5e519940eb9c1254c8d9122846f4d45ff45c1f not found: ID does not exist"
Sep 08 18:56:55.994440 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:55.994387    2579 scope.go:117] "RemoveContainer" containerID="ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b"
Sep 08 18:56:56.005575 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.005554    2579 scope.go:117] "RemoveContainer" containerID="ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b"
Sep 08 18:56:56.005885 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:56:56.005867    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b\": container with ID starting with ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b not found: ID does not exist" containerID="ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b"
Sep 08 18:56:56.005938 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.005892    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b"} err="failed to get container status \"ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b\": rpc error: code = NotFound desc = could not find container \"ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b\": container with ID starting with ea1c6bb11bb5c9bd33ec484c2e2c18145ab7f3d4c6c9fac7d30d341d792de19b not found: ID does not exist"
Sep 08 18:56:56.005938 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.005907    2579 scope.go:117] "RemoveContainer" containerID="7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69"
Sep 08 18:56:56.012232 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.012212    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"]
Sep 08 18:56:56.014307 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.014288    2579 scope.go:117] "RemoveContainer" containerID="7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69"
Sep 08 18:56:56.014620 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:56:56.014600    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69\": container with ID starting with 7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69 not found: ID does not exist" containerID="7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69"
Sep 08 18:56:56.014678 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.014627    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69"} err="failed to get container status \"7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69\": rpc error: code = NotFound desc = could not find container \"7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69\": container with ID starting with 7454d9fa62900ef389ebced769821f856e8d208a8cbcd4d4df3bc68646868f69 not found: ID does not exist"
Sep 08 18:56:56.015831 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.015814    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-64effff4-74fffd9878-7vk8z"]
Sep 08 18:56:56.027139 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.027117    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"]
Sep 08 18:56:56.033595 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.033575    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-64effff4-84f89f7d9c-lkfrd"]
Sep 08 18:56:56.045355 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.045316    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"]
Sep 08 18:56:56.049070 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.049049    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-64effff4-565cfcbb94-6v2lt"]
Sep 08 18:56:56.163373 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.163274    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d993a9-09ed-45ae-aea5-ab12ab32d005" path="/var/lib/kubelet/pods/23d993a9-09ed-45ae-aea5-ab12ab32d005/volumes"
Sep 08 18:56:56.163688 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.163673    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c7694e2-3a3d-41f9-9123-899fc00ab6a9" path="/var/lib/kubelet/pods/4c7694e2-3a3d-41f9-9123-899fc00ab6a9/volumes"
Sep 08 18:56:56.164019 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.164008    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a68f3c-1629-4f4e-837d-511707220f45" path="/var/lib/kubelet/pods/83a68f3c-1629-4f4e-837d-511707220f45/volumes"
Sep 08 18:56:56.236616 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.236589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:56.688220 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:56.688183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:59.194785 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:59.194748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:56:59.293540 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:56:59.293501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:00.287947 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.287917    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"]
Sep 08 18:57:00.288489 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288462    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a68f3c-1629-4f4e-837d-511707220f45" containerName="maas-api"
Sep 08 18:57:00.288539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288492    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a68f3c-1629-4f4e-837d-511707220f45" containerName="maas-api"
Sep 08 18:57:00.288539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288512    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="304461d6-51d5-4029-88a4-c049f1e6a873" containerName="main"
Sep 08 18:57:00.288539 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288520    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="304461d6-51d5-4029-88a4-c049f1e6a873" containerName="main"
Sep 08 18:57:00.288634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288539    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7694e2-3a3d-41f9-9123-899fc00ab6a9" containerName="payload-pre-processing"
Sep 08 18:57:00.288634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288549    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7694e2-3a3d-41f9-9123-899fc00ab6a9" containerName="payload-pre-processing"
Sep 08 18:57:00.288634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288573    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a33aabc-df48-4975-893c-b9d6e073ea13" containerName="main"
Sep 08 18:57:00.288634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288581    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a33aabc-df48-4975-893c-b9d6e073ea13" containerName="main"
Sep 08 18:57:00.288634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288594    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d993a9-09ed-45ae-aea5-ab12ab32d005" containerName="payload-processing"
Sep 08 18:57:00.288634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288603    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d993a9-09ed-45ae-aea5-ab12ab32d005" containerName="payload-processing"
Sep 08 18:57:00.288634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288623    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce7893f-9c93-47a6-8225-ed9f17170702" containerName="main"
Sep 08 18:57:00.288634 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288631    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce7893f-9c93-47a6-8225-ed9f17170702" containerName="main"
Sep 08 18:57:00.288862 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288720    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7694e2-3a3d-41f9-9123-899fc00ab6a9" containerName="payload-pre-processing"
Sep 08 18:57:00.288862 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288735    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a68f3c-1629-4f4e-837d-511707220f45" containerName="maas-api"
Sep 08 18:57:00.288862 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288743    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="304461d6-51d5-4029-88a4-c049f1e6a873" containerName="main"
Sep 08 18:57:00.288862 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288755    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce7893f-9c93-47a6-8225-ed9f17170702" containerName="main"
Sep 08 18:57:00.288862 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288763    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a33aabc-df48-4975-893c-b9d6e073ea13" containerName="main"
Sep 08 18:57:00.288862 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.288772    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d993a9-09ed-45ae-aea5-ab12ab32d005" containerName="payload-processing"
Sep 08 18:57:00.293652 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.293620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.296329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.296300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a4001b-serving-cert\""
Sep 08 18:57:00.296447 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.296309    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:57:00.296447 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.296314    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:57:00.296447 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.296387    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:57:00.296447 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.296329    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 18:57:00.300179 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.300150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"]
Sep 08 18:57:00.395475 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.395435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.395674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.395548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbnh\" (UniqueName: \"kubernetes.io/projected/c9af8476-17bb-4bf4-8160-561042ccb373-kube-api-access-5xbnh\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.395674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.395591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.395674 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.395620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af8476-17bb-4bf4-8160-561042ccb373-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.435750 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.435716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:00.496787 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.496705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbnh\" (UniqueName: \"kubernetes.io/projected/c9af8476-17bb-4bf4-8160-561042ccb373-kube-api-access-5xbnh\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.496787 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.496751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.496787 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.496773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af8476-17bb-4bf4-8160-561042ccb373-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.497032 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.496812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.497032 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:57:00.496865    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-serving-cert: secret "maas-api-e2e-shared-a-w3-a4001b-serving-cert" not found
Sep 08 18:57:00.497032 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:57:00.496952    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls podName:c9af8476-17bb-4bf4-8160-561042ccb373 nodeName:}" failed. No retries permitted until 2026-09-08 18:57:00.996929132 +0000 UTC m=+1381.351022458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq" (UID: "c9af8476-17bb-4bf4-8160-561042ccb373") : secret "maas-api-e2e-shared-a-w3-a4001b-serving-cert" not found
Sep 08 18:57:00.497526 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.497499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af8476-17bb-4bf4-8160-561042ccb373-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.499134 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.499117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.504867 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.504845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbnh\" (UniqueName: \"kubernetes.io/projected/c9af8476-17bb-4bf4-8160-561042ccb373-kube-api-access-5xbnh\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:00.543568 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.543539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:00.827455 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.827363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:00.932554 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:00.932520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:01.001884 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.001847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:01.004106 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.004087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:01.031489 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.031457    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"]
Sep 08 18:57:01.035001 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.034985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:01.037707 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.037686    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a4001b-dockercfg-h82r6\""
Sep 08 18:57:01.038279 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.038262    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a4001b\""
Sep 08 18:57:01.044425 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.044399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"]
Sep 08 18:57:01.102938 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.102911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh\" (UID: \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:01.103098 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.102961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88n8\" (UniqueName: \"kubernetes.io/projected/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-kube-api-access-x88n8\") pod \"payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh\" (UID: \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:01.126609 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.126588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:01.204110 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.204076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh\" (UID: \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:01.204265 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.204129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88n8\" (UniqueName: \"kubernetes.io/projected/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-kube-api-access-x88n8\") pod \"payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh\" (UID: \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:01.204866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.204845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh\" (UID: \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:01.205913 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.205893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:01.214872 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.214851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88n8\" (UniqueName: \"kubernetes.io/projected/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-kube-api-access-x88n8\") pod \"payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh\" (UID: \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:01.346285 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.346245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:01.475621 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.475596    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"]
Sep 08 18:57:01.477947 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:57:01.477915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9af8476_17bb_4bf4_8160_561042ccb373.slice/crio-0a2319609c3acb6ef200984a66b90090389b6c14ae7e7cea9acfa181c6931058 WatchSource:0}: Error finding container 0a2319609c3acb6ef200984a66b90090389b6c14ae7e7cea9acfa181c6931058: Status 404 returned error can't find the container with id 0a2319609c3acb6ef200984a66b90090389b6c14ae7e7cea9acfa181c6931058
Sep 08 18:57:01.615961 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:01.615933    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"]
Sep 08 18:57:01.618468 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:57:01.618440    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65fa616_2411_4ea8_9fdc_6fd33b0e698a.slice/crio-3c1e178b1d24022f06dd1eee731ee6eda94afc117834c934cb8b2a17284d6e55 WatchSource:0}: Error finding container 3c1e178b1d24022f06dd1eee731ee6eda94afc117834c934cb8b2a17284d6e55: Status 404 returned error can't find the container with id 3c1e178b1d24022f06dd1eee731ee6eda94afc117834c934cb8b2a17284d6e55
Sep 08 18:57:02.022687 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.022645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh" event={"ID":"b65fa616-2411-4ea8-9fdc-6fd33b0e698a","Type":"ContainerStarted","Data":"70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63"}
Sep 08 18:57:02.022864 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.022718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh" event={"ID":"b65fa616-2411-4ea8-9fdc-6fd33b0e698a","Type":"ContainerStarted","Data":"3c1e178b1d24022f06dd1eee731ee6eda94afc117834c934cb8b2a17284d6e55"}
Sep 08 18:57:02.022864 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.022751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:02.024447 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.024409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq" event={"ID":"c9af8476-17bb-4bf4-8160-561042ccb373","Type":"ContainerStarted","Data":"abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95"}
Sep 08 18:57:02.024594 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.024460    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq" event={"ID":"c9af8476-17bb-4bf4-8160-561042ccb373","Type":"ContainerStarted","Data":"0a2319609c3acb6ef200984a66b90090389b6c14ae7e7cea9acfa181c6931058"}
Sep 08 18:57:02.024594 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.024513    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:02.041545 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.041498    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh" podStartSLOduration=1.041481608 podStartE2EDuration="1.041481608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:02.041322029 +0000 UTC m=+1382.395415374" watchObservedRunningTime="2026-09-08 18:57:02.041481608 +0000 UTC m=+1382.395574940"
Sep 08 18:57:02.059714 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.059658    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq" podStartSLOduration=1.702746797 podStartE2EDuration="2.059642151s" totalImagesPullingTime="356.895354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:02.057608841 +0000 UTC m=+1382.411702174" watchObservedRunningTime="2026-09-08 18:57:02.059642151 +0000 UTC m=+1382.413735483"
Sep 08 18:57:02.073478 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.073448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:02.177154 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.177057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:02.341701 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.341665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:02.740054 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.740015    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:02.783527 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.783497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:57:02.842707 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.842679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:02.969731 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:02.969702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:08.040293 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:08.040254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 18:57:08.896853 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:08.896819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:09.001542 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:09.001506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:09.096293 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:09.096257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:09.251491 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:09.251409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:13.037146 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:13.037115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 18:57:14.973562 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:14.973527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:15.044781 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:15.044747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:15.922175 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:15.922142    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:16.285703 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:16.285622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:16.451498 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:16.451463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:19.202246 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:19.202200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:19.291637 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:19.291598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:20.348392 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:20.348353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:20.452551 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:20.452518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:20.568454 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:20.568422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:20.925980 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:20.925951    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:21.044869 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.044830    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:21.045830 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.045810    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"]
Sep 08 18:57:21.050756 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.050737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:21.055382 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.055360    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-093570\""
Sep 08 18:57:21.055500 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.055384    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-093570-dockercfg-z2fh8\""
Sep 08 18:57:21.069810 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.069789    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"]
Sep 08 18:57:21.193849 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.193775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779392ca-78f1-4bf0-9518-5909cc02efb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s\" (UID: \"779392ca-78f1-4bf0-9518-5909cc02efb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:21.193849 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.193825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6mkc\" (UniqueName: \"kubernetes.io/projected/779392ca-78f1-4bf0-9518-5909cc02efb1-kube-api-access-c6mkc\") pod \"payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s\" (UID: \"779392ca-78f1-4bf0-9518-5909cc02efb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:21.295244 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.294838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779392ca-78f1-4bf0-9518-5909cc02efb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s\" (UID: \"779392ca-78f1-4bf0-9518-5909cc02efb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:21.295244 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.295210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6mkc\" (UniqueName: \"kubernetes.io/projected/779392ca-78f1-4bf0-9518-5909cc02efb1-kube-api-access-c6mkc\") pod \"payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s\" (UID: \"779392ca-78f1-4bf0-9518-5909cc02efb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:21.295747 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.295727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779392ca-78f1-4bf0-9518-5909cc02efb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s\" (UID: \"779392ca-78f1-4bf0-9518-5909cc02efb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:21.303090 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.303067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6mkc\" (UniqueName: \"kubernetes.io/projected/779392ca-78f1-4bf0-9518-5909cc02efb1-kube-api-access-c6mkc\") pod \"payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s\" (UID: \"779392ca-78f1-4bf0-9518-5909cc02efb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:21.360468 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.360433    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:21.627533 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:21.627473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"]
Sep 08 18:57:21.630098 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:57:21.630060    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779392ca_78f1_4bf0_9518_5909cc02efb1.slice/crio-58255c23764b173c1e68db1cb3dcbd48ee11f5976450abd903c1af586b03004b WatchSource:0}: Error finding container 58255c23764b173c1e68db1cb3dcbd48ee11f5976450abd903c1af586b03004b: Status 404 returned error can't find the container with id 58255c23764b173c1e68db1cb3dcbd48ee11f5976450abd903c1af586b03004b
Sep 08 18:57:22.121693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:22.121657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s" event={"ID":"779392ca-78f1-4bf0-9518-5909cc02efb1","Type":"ContainerStarted","Data":"a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f"}
Sep 08 18:57:22.121840 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:22.121718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s" event={"ID":"779392ca-78f1-4bf0-9518-5909cc02efb1","Type":"ContainerStarted","Data":"58255c23764b173c1e68db1cb3dcbd48ee11f5976450abd903c1af586b03004b"}
Sep 08 18:57:22.121840 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:22.121741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:22.160419 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:22.160373    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s" podStartSLOduration=1.160356196 podStartE2EDuration="1.160356196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:22.159086621 +0000 UTC m=+1402.513179955" watchObservedRunningTime="2026-09-08 18:57:22.160356196 +0000 UTC m=+1402.514449526"
Sep 08 18:57:22.501132 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:22.501052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:24.198227 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:24.198188    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:24.532315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:24.532227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:24.838171 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:24.838138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:24.994946 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:24.994910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:25.174370 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:25.174262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:57:28.371607 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:28.371576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:28.674793 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:28.674703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:29.479193 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.479162    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"]
Sep 08 18:57:29.483215 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.483188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.485750 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.485729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-525196-serving-cert\""
Sep 08 18:57:29.495290 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.495262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"]
Sep 08 18:57:29.525944 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.525910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:29.566300 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.566268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mlh\" (UniqueName: \"kubernetes.io/projected/08482b49-d555-47f9-86c3-41ffad57416e-kube-api-access-n8mlh\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.566471 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.566329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.566471 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.566416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08482b49-d555-47f9-86c3-41ffad57416e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.566471 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.566458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.667552 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.667516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.667695 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.667596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mlh\" (UniqueName: \"kubernetes.io/projected/08482b49-d555-47f9-86c3-41ffad57416e-kube-api-access-n8mlh\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.667695 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.667644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.667695 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.667678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08482b49-d555-47f9-86c3-41ffad57416e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.668280 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.668255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08482b49-d555-47f9-86c3-41ffad57416e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.669981 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.669954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.670062 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.669999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.674488 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.674467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mlh\" (UniqueName: \"kubernetes.io/projected/08482b49-d555-47f9-86c3-41ffad57416e-kube-api-access-n8mlh\") pod \"maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.794796 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.794712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:29.861548 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.861520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:29.963012 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:29.962980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:30.066760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:30.066672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:30.067751 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:30.067728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"]
Sep 08 18:57:30.070396 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:57:30.070373    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08482b49_d555_47f9_86c3_41ffad57416e.slice/crio-a29dad42ada384c8e1b94e88f3b750068d945d896d31e3f90dc707d50ee729ff WatchSource:0}: Error finding container a29dad42ada384c8e1b94e88f3b750068d945d896d31e3f90dc707d50ee729ff: Status 404 returned error can't find the container with id a29dad42ada384c8e1b94e88f3b750068d945d896d31e3f90dc707d50ee729ff
Sep 08 18:57:30.146914 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:30.146885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:30.165005 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:30.164978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v" event={"ID":"08482b49-d555-47f9-86c3-41ffad57416e","Type":"ContainerStarted","Data":"a29dad42ada384c8e1b94e88f3b750068d945d896d31e3f90dc707d50ee729ff"}
Sep 08 18:57:31.172329 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:31.172290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v" event={"ID":"08482b49-d555-47f9-86c3-41ffad57416e","Type":"ContainerStarted","Data":"d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b"}
Sep 08 18:57:31.172728 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:31.172369    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:31.189527 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:31.189481    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v" podStartSLOduration=1.7870701979999999 podStartE2EDuration="2.189467698s" totalImagesPullingTime="402.3975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:31.189144823 +0000 UTC m=+1411.543238156" watchObservedRunningTime="2026-09-08 18:57:31.189467698 +0000 UTC m=+1411.543561030"
Sep 08 18:57:33.134261 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:33.134226    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 18:57:35.680358 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:35.680314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:57:37.182461 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:37.182429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 18:57:41.342269 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:41.342238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:44.261207 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:44.261174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:44.614502 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:44.614468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:44.948019 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:44.947935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:45.290885 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:45.290806    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:48.526030 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:48.525983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:48.643058 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:48.643025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:49.579720 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.579686    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"]
Sep 08 18:57:49.585088 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.585063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.587544 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.587522    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c6f2ed-serving-cert\""
Sep 08 18:57:49.591965 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.591936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"]
Sep 08 18:57:49.614212 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.614180    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:49.652012 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.651985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbckd\" (UniqueName: \"kubernetes.io/projected/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-kube-api-access-kbckd\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.652137 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.652033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.652137 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.652068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.652137 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.652103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.715527 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.715503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:49.753297 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.753262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.753297 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.753300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.753535 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.753329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.753535 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:57:49.753430    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-serving-cert: secret "maas-api-e2e-shared-b-w2-c6f2ed-serving-cert" not found
Sep 08 18:57:49.753535 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.753484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbckd\" (UniqueName: \"kubernetes.io/projected/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-kube-api-access-kbckd\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.753535 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:57:49.753507    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls podName:c5c56237-1ea3-4da6-ae4f-92c54f3b87f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:57:50.253485531 +0000 UTC m=+1430.607578858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6" (UID: "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9") : secret "maas-api-e2e-shared-b-w2-c6f2ed-serving-cert" not found
Sep 08 18:57:49.754109 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.754085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.755710 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.755691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:49.761408 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:49.761388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbckd\" (UniqueName: \"kubernetes.io/projected/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-kube-api-access-kbckd\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:50.121513 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.121481    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:50.215767 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.215733    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"]
Sep 08 18:57:50.219497 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.219480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:50.221714 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.221683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c6f2ed-dockercfg-j66pk\""
Sep 08 18:57:50.221852 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.221731    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c6f2ed\""
Sep 08 18:57:50.221994 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.221975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:50.230711 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.230687    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"]
Sep 08 18:57:50.258650 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.258623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:50.261011 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.260986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:50.359252 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.359211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea635718-2e6e-44b2-b256-bd909bbddc41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d\" (UID: \"ea635718-2e6e-44b2-b256-bd909bbddc41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:50.359444 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.359314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2wc\" (UniqueName: \"kubernetes.io/projected/ea635718-2e6e-44b2-b256-bd909bbddc41-kube-api-access-nw2wc\") pod \"payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d\" (UID: \"ea635718-2e6e-44b2-b256-bd909bbddc41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:50.460653 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.460564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea635718-2e6e-44b2-b256-bd909bbddc41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d\" (UID: \"ea635718-2e6e-44b2-b256-bd909bbddc41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:50.460653 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.460634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2wc\" (UniqueName: \"kubernetes.io/projected/ea635718-2e6e-44b2-b256-bd909bbddc41-kube-api-access-nw2wc\") pod \"payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d\" (UID: \"ea635718-2e6e-44b2-b256-bd909bbddc41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:50.461362 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.461318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea635718-2e6e-44b2-b256-bd909bbddc41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d\" (UID: \"ea635718-2e6e-44b2-b256-bd909bbddc41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:50.469207 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.469176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2wc\" (UniqueName: \"kubernetes.io/projected/ea635718-2e6e-44b2-b256-bd909bbddc41-kube-api-access-nw2wc\") pod \"payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d\" (UID: \"ea635718-2e6e-44b2-b256-bd909bbddc41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:50.498960 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.498928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:50.531004 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.530968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:50.647699 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.647662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:50.773002 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.772980    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"]
Sep 08 18:57:50.775310 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:57:50.775283    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c56237_1ea3_4da6_ae4f_92c54f3b87f9.slice/crio-61d47763f19a51c931b7e2285b129e48d67b62b84ebd6653ddb9f07f02643df1 WatchSource:0}: Error finding container 61d47763f19a51c931b7e2285b129e48d67b62b84ebd6653ddb9f07f02643df1: Status 404 returned error can't find the container with id 61d47763f19a51c931b7e2285b129e48d67b62b84ebd6653ddb9f07f02643df1
Sep 08 18:57:50.810743 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:50.810723    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"]
Sep 08 18:57:50.812697 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:57:50.812666    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea635718_2e6e_44b2_b256_bd909bbddc41.slice/crio-0d029c4bc392633f7379beccdc1611d49f3ac670954347512b32fa712d480695 WatchSource:0}: Error finding container 0d029c4bc392633f7379beccdc1611d49f3ac670954347512b32fa712d480695: Status 404 returned error can't find the container with id 0d029c4bc392633f7379beccdc1611d49f3ac670954347512b32fa712d480695
Sep 08 18:57:51.263796 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.263734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d" event={"ID":"ea635718-2e6e-44b2-b256-bd909bbddc41","Type":"ContainerStarted","Data":"144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a"}
Sep 08 18:57:51.263971 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.263817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:57:51.263971 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.263843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d" event={"ID":"ea635718-2e6e-44b2-b256-bd909bbddc41","Type":"ContainerStarted","Data":"0d029c4bc392633f7379beccdc1611d49f3ac670954347512b32fa712d480695"}
Sep 08 18:57:51.265636 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.265607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6" event={"ID":"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9","Type":"ContainerStarted","Data":"f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7"}
Sep 08 18:57:51.265760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.265650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6" event={"ID":"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9","Type":"ContainerStarted","Data":"61d47763f19a51c931b7e2285b129e48d67b62b84ebd6653ddb9f07f02643df1"}
Sep 08 18:57:51.265760 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.265684    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:57:51.284600 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.284560    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d" podStartSLOduration=1.28454449 podStartE2EDuration="1.28454449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:51.28343808 +0000 UTC m=+1431.637531428" watchObservedRunningTime="2026-09-08 18:57:51.28454449 +0000 UTC m=+1431.638637822"
Sep 08 18:57:51.306605 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.306561    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6" podStartSLOduration=1.925332896 podStartE2EDuration="2.306546407s" totalImagesPullingTime="381.213511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:51.304923726 +0000 UTC m=+1431.659017060" watchObservedRunningTime="2026-09-08 18:57:51.306546407 +0000 UTC m=+1431.660639739"
Sep 08 18:57:51.550312 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:51.550218    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:52.231904 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:52.231875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:53.325315 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:53.325283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:57:56.178278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:56.178241    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:57:57.276024 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:57:57.275991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 18:58:02.272866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:02.272832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 18:58:11.393361 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:11.393316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:13.518299 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:13.518265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:18.432543 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:18.432507    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7692-7f1f56"]
Sep 08 18:58:18.437378 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:18.437292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-7f1f56"
Sep 08 18:58:18.439519 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:18.439493    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgjvd\""
Sep 08 18:58:18.439670 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:18.439556    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-7f1f56"]
Sep 08 18:58:18.519246 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:18.519209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6db\" (UniqueName: \"kubernetes.io/projected/deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2-kube-api-access-9x6db\") pod \"test-curl-7692-7f1f56\" (UID: \"deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2\") " pod="opendatahub/test-curl-7692-7f1f56"
Sep 08 18:58:18.620181 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:18.620138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6db\" (UniqueName: \"kubernetes.io/projected/deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2-kube-api-access-9x6db\") pod \"test-curl-7692-7f1f56\" (UID: \"deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2\") " pod="opendatahub/test-curl-7692-7f1f56"
Sep 08 18:58:18.628062 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:18.628038    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6db\" (UniqueName: \"kubernetes.io/projected/deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2-kube-api-access-9x6db\") pod \"test-curl-7692-7f1f56\" (UID: \"deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2\") " pod="opendatahub/test-curl-7692-7f1f56"
Sep 08 18:58:18.749965 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:18.749882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-7f1f56"
Sep 08 18:58:19.021557 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:19.021534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-7f1f56"]
Sep 08 18:58:19.024047 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:58:19.024020    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeafcf96_f77a_4df3_aa5b_1de4b7e1ffa2.slice/crio-7b7f76ed5b6e9f2537275f2255d6f20d9bbcce8e138160d287e8b82ab1fa0035 WatchSource:0}: Error finding container 7b7f76ed5b6e9f2537275f2255d6f20d9bbcce8e138160d287e8b82ab1fa0035: Status 404 returned error can't find the container with id 7b7f76ed5b6e9f2537275f2255d6f20d9bbcce8e138160d287e8b82ab1fa0035
Sep 08 18:58:19.412084 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:19.412048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-7f1f56" event={"ID":"deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2","Type":"ContainerStarted","Data":"77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e"}
Sep 08 18:58:19.412257 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:19.412091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-7f1f56" event={"ID":"deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2","Type":"ContainerStarted","Data":"7b7f76ed5b6e9f2537275f2255d6f20d9bbcce8e138160d287e8b82ab1fa0035"}
Sep 08 18:58:19.427044 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:19.426994    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7692-7f1f56" podStartSLOduration=1.42697809 podStartE2EDuration="1.42697809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:19.424994377 +0000 UTC m=+1459.779087733" watchObservedRunningTime="2026-09-08 18:58:19.42697809 +0000 UTC m=+1459.781071423"
Sep 08 18:58:20.056995 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:20.056957    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7692-7f1f56"]
Sep 08 18:58:20.061785 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:20.061759    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7692-7f1f56"]
Sep 08 18:58:20.416586 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:58:20.416555    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7692-7f1f56" container="test-curl-7692-7f1f56"
Sep 08 18:58:21.420712 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:21.420667    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7692-7f1f56" podUID="deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2" containerName="test-curl-7692-7f1f56" containerID="cri-o://77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e" gracePeriod=2
Sep 08 18:58:22.952976 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:22.952908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:23.288854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:23.288781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:23.596313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:23.596280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-7f1f56"
Sep 08 18:58:23.667178 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:23.667147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x6db\" (UniqueName: \"kubernetes.io/projected/deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2-kube-api-access-9x6db\") pod \"deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2\" (UID: \"deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2\") "
Sep 08 18:58:23.669179 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:23.669157    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2-kube-api-access-9x6db" (OuterVolumeSpecName: "kube-api-access-9x6db") pod "deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2" (UID: "deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2"). InnerVolumeSpecName "kube-api-access-9x6db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:58:23.761529 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:23.761495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:23.768801 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:23.768775    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x6db\" (UniqueName: \"kubernetes.io/projected/deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2-kube-api-access-9x6db\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 18:58:24.163004 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:24.162957    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2" path="/var/lib/kubelet/pods/deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2/volumes"
Sep 08 18:58:24.437269 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:24.437174    2579 generic.go:356] "Generic (PLEG): container finished" podID="deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2" containerID="77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e" exitCode=137
Sep 08 18:58:24.437269 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:24.437215    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-7f1f56"
Sep 08 18:58:24.437518 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:24.437274    2579 scope.go:117] "RemoveContainer" containerID="77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e"
Sep 08 18:58:24.445592 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:24.445576    2579 scope.go:117] "RemoveContainer" containerID="77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e"
Sep 08 18:58:24.445866 ip-10-0-140-26 kubenswrapper[2579]: E0908 18:58:24.445846    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e\": container with ID starting with 77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e not found: ID does not exist" containerID="77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e"
Sep 08 18:58:24.445914 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:24.445875    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e"} err="failed to get container status \"77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e\": rpc error: code = NotFound desc = could not find container \"77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e\": container with ID starting with 77b5bb910aac96792c172378d32300152591676bf9b5a0b54268093715c6316e not found: ID does not exist"
Sep 08 18:58:26.782455 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:26.782415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:26.887234 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:26.887204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:28.054286 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.054258    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:28.128962 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.128932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:28.271277 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.271241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:28.642227 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.642195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:28.744814 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.744785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:28.790574 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.790547    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"]
Sep 08 18:58:28.790927 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.790913    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2" containerName="test-curl-7692-7f1f56"
Sep 08 18:58:28.790976 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.790928    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2" containerName="test-curl-7692-7f1f56"
Sep 08 18:58:28.791011 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.791003    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="deafcf96-f77a-4df3-aa5b-1de4b7e1ffa2" containerName="test-curl-7692-7f1f56"
Sep 08 18:58:28.795527 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.795512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:28.798511 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.798492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3adca991-dockercfg-dghmk\""
Sep 08 18:58:28.799333 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.799311    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3adca991\""
Sep 08 18:58:28.816395 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.816369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"]
Sep 08 18:58:28.835822 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.835797    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"]
Sep 08 18:58:28.839323 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.839306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:28.852058 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.852033    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"]
Sep 08 18:58:28.916866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.916794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-plugins-config-volume\") pod \"payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5\" (UID: \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\") " pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:28.916866 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.916836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhzn\" (UniqueName: \"kubernetes.io/projected/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-kube-api-access-6xhzn\") pod \"payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5\" (UID: \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\") " pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:28.917032 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.916885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gj2d\" (UniqueName: \"kubernetes.io/projected/26a29991-18b5-4bed-aa5e-30a8fa23cad0-kube-api-access-6gj2d\") pod \"payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k\" (UID: \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:28.917032 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:28.916949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a29991-18b5-4bed-aa5e-30a8fa23cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k\" (UID: \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:29.017524 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.017483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gj2d\" (UniqueName: \"kubernetes.io/projected/26a29991-18b5-4bed-aa5e-30a8fa23cad0-kube-api-access-6gj2d\") pod \"payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k\" (UID: \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:29.017721 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.017550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a29991-18b5-4bed-aa5e-30a8fa23cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k\" (UID: \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:29.017721 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.017621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-plugins-config-volume\") pod \"payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5\" (UID: \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\") " pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:29.017721 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.017651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhzn\" (UniqueName: \"kubernetes.io/projected/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-kube-api-access-6xhzn\") pod \"payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5\" (UID: \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\") " pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:29.018284 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.018261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a29991-18b5-4bed-aa5e-30a8fa23cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k\" (UID: \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:29.018374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.018330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-plugins-config-volume\") pod \"payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5\" (UID: \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\") " pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:29.026196 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.026170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gj2d\" (UniqueName: \"kubernetes.io/projected/26a29991-18b5-4bed-aa5e-30a8fa23cad0-kube-api-access-6gj2d\") pod \"payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k\" (UID: \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:29.026196 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.026190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhzn\" (UniqueName: \"kubernetes.io/projected/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-kube-api-access-6xhzn\") pod \"payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5\" (UID: \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\") " pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:29.105423 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.105394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:29.149545 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.149514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:29.384088 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.384064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"]
Sep 08 18:58:29.386404 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:58:29.386379    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398ac735_8b2f_48ce_9c6a_a6b9d637b53b.slice/crio-351d73e19443b25dbffb8fac82dc1348946df64ae1a1dee8aa6504e3cf4f2356 WatchSource:0}: Error finding container 351d73e19443b25dbffb8fac82dc1348946df64ae1a1dee8aa6504e3cf4f2356: Status 404 returned error can't find the container with id 351d73e19443b25dbffb8fac82dc1348946df64ae1a1dee8aa6504e3cf4f2356
Sep 08 18:58:29.432297 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.432263    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"]
Sep 08 18:58:29.434788 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:58:29.434768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a29991_18b5_4bed_aa5e_30a8fa23cad0.slice/crio-64a89c3cd73930f83935f1cd5f50ab7d1bc88a110f5a4ed0991e3fc9b4b2c8f6 WatchSource:0}: Error finding container 64a89c3cd73930f83935f1cd5f50ab7d1bc88a110f5a4ed0991e3fc9b4b2c8f6: Status 404 returned error can't find the container with id 64a89c3cd73930f83935f1cd5f50ab7d1bc88a110f5a4ed0991e3fc9b4b2c8f6
Sep 08 18:58:29.462369 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.462313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k" event={"ID":"26a29991-18b5-4bed-aa5e-30a8fa23cad0","Type":"ContainerStarted","Data":"64a89c3cd73930f83935f1cd5f50ab7d1bc88a110f5a4ed0991e3fc9b4b2c8f6"}
Sep 08 18:58:29.463947 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.463921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5" event={"ID":"398ac735-8b2f-48ce-9c6a-a6b9d637b53b","Type":"ContainerStarted","Data":"a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe"}
Sep 08 18:58:29.464062 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.463970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5" event={"ID":"398ac735-8b2f-48ce-9c6a-a6b9d637b53b","Type":"ContainerStarted","Data":"351d73e19443b25dbffb8fac82dc1348946df64ae1a1dee8aa6504e3cf4f2356"}
Sep 08 18:58:29.464130 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.464055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:29.493876 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:29.493831    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5" podStartSLOduration=1.49380948 podStartE2EDuration="1.49380948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:29.49212046 +0000 UTC m=+1469.846213795" watchObservedRunningTime="2026-09-08 18:58:29.49380948 +0000 UTC m=+1469.847902812"
Sep 08 18:58:30.262355 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.262315    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"]
Sep 08 18:58:30.265758 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.265741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.267885 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.267864    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a4001b\"/\"kube-root-ca.crt\""
Sep 08 18:58:30.268552 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.268529    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a4001b\"/\"openshift-service-ca.crt\""
Sep 08 18:58:30.268693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.268536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a4001b\"/\"default-dockercfg-8gfv5\""
Sep 08 18:58:30.268693 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.268536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a4001b\"/\"auth-test-model-a4001b-kserve-self-signed-certs\""
Sep 08 18:58:30.274894 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.274867    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"]
Sep 08 18:58:30.330620 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.330591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-dshm\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.330620 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.330625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-model-cache\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.330854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.330656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-home\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.330854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.330793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8td\" (UniqueName: \"kubernetes.io/projected/5621170d-06d6-44a8-bad0-72015e2ae9d8-kube-api-access-vj8td\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.330854 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.330837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5621170d-06d6-44a8-bad0-72015e2ae9d8-tls-certs\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.331006 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.330912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-tmp-dir\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.431422 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.431392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8td\" (UniqueName: \"kubernetes.io/projected/5621170d-06d6-44a8-bad0-72015e2ae9d8-kube-api-access-vj8td\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.431617 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.431438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5621170d-06d6-44a8-bad0-72015e2ae9d8-tls-certs\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.431617 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.431493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-tmp-dir\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.431617 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.431549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-dshm\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.431617 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.431573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-model-cache\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.431617 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.431604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-home\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.432045 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.432020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-model-cache\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.432155 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.432074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-home\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.432221 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.432157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-tmp-dir\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.434072 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.434045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-dshm\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.434196 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.434167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5621170d-06d6-44a8-bad0-72015e2ae9d8-tls-certs\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.439743 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.439721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8td\" (UniqueName: \"kubernetes.io/projected/5621170d-06d6-44a8-bad0-72015e2ae9d8-kube-api-access-vj8td\") pod \"auth-test-model-a4001b-kserve-cb5568475-n685l\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.470313 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.470273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k" event={"ID":"26a29991-18b5-4bed-aa5e-30a8fa23cad0","Type":"ContainerStarted","Data":"3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c"}
Sep 08 18:58:30.470540 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.470520    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:30.492808 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.492740    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k" podStartSLOduration=2.492719898 podStartE2EDuration="2.492719898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:30.490823139 +0000 UTC m=+1470.844916472" watchObservedRunningTime="2026-09-08 18:58:30.492719898 +0000 UTC m=+1470.846813235"
Sep 08 18:58:30.578220 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.578133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:30.675724 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.675686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:30.859171 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:30.859127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"]
Sep 08 18:58:30.862198 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:58:30.862169    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5621170d_06d6_44a8_bad0_72015e2ae9d8.slice/crio-a705585b09a2f2f09de1088b9024b539149f6099a937bc39baccb633a91ef7c4 WatchSource:0}: Error finding container a705585b09a2f2f09de1088b9024b539149f6099a937bc39baccb633a91ef7c4: Status 404 returned error can't find the container with id a705585b09a2f2f09de1088b9024b539149f6099a937bc39baccb633a91ef7c4
Sep 08 18:58:31.477568 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:31.477536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l" event={"ID":"5621170d-06d6-44a8-bad0-72015e2ae9d8","Type":"ContainerStarted","Data":"1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0"}
Sep 08 18:58:31.478033 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:31.477582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l" event={"ID":"5621170d-06d6-44a8-bad0-72015e2ae9d8","Type":"ContainerStarted","Data":"a705585b09a2f2f09de1088b9024b539149f6099a937bc39baccb633a91ef7c4"}
Sep 08 18:58:31.494942 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:31.494891    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l" podStartSLOduration=1.494875023 podStartE2EDuration="1.494875023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:31.494314 +0000 UTC m=+1471.848407403" watchObservedRunningTime="2026-09-08 18:58:31.494875023 +0000 UTC m=+1471.848968355"
Sep 08 18:58:36.476706 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:36.476669    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:58:40.472805 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:40.472774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 18:58:40.578446 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:40.578397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:40.578616 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:40.578462    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:40.591211 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:40.591183    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:41.480219 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:41.480192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 18:58:41.538275 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:41.538247    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 18:58:43.837100 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:43.837064    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:51.838016 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:51.837983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:58:54.875940 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:58:54.875910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:59:00.186732 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:00.186707    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:59:00.190411 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:00.190387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 18:59:04.175982 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.175949    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"]
Sep 08 18:59:04.181002 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.180980    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.183256 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.183235    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"kube-root-ca.crt\""
Sep 08 18:59:04.183837 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.183817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"openshift-service-ca.crt\""
Sep 08 18:59:04.184176 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.184159    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"default-dockercfg-tbv89\""
Sep 08 18:59:04.184243 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.184185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"auth-test-model-525196-kserve-self-signed-certs\""
Sep 08 18:59:04.189180 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.189157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"]
Sep 08 18:59:04.199953 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.199927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:59:04.256553 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.256523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4lc\" (UniqueName: \"kubernetes.io/projected/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-kube-api-access-7v4lc\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.256672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.256573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-model-cache\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.256672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.256600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tmp-dir\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.256672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.256627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-home\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.256672 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.256661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tls-certs\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.256812 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.256730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-dshm\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.357694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.357664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-model-cache\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.357694 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.357698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tmp-dir\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.357899 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.357723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-home\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.357899 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.357758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tls-certs\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.357899 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.357776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-dshm\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.357899 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.357833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4lc\" (UniqueName: \"kubernetes.io/projected/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-kube-api-access-7v4lc\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.358185 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.358018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-model-cache\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.358185 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.358092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tmp-dir\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.358185 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.358167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-home\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.360025 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.359992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-dshm\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.360251 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.360232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tls-certs\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.365142 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.365112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4lc\" (UniqueName: \"kubernetes.io/projected/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-kube-api-access-7v4lc\") pod \"auth-test-model-525196-kserve-749fb46699-kqv6v\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") " pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.492076 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.491983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:04.762595 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:04.762572    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"]
Sep 08 18:59:04.764325 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:59:04.764298    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcea4e9e3_ee9e_43a8_8a90_7f3af28ba87a.slice/crio-0934c557e4f45b7fc2a56a27f1a1a362253e527ab8c2e21208d380a158a17179 WatchSource:0}: Error finding container 0934c557e4f45b7fc2a56a27f1a1a362253e527ab8c2e21208d380a158a17179: Status 404 returned error can't find the container with id 0934c557e4f45b7fc2a56a27f1a1a362253e527ab8c2e21208d380a158a17179
Sep 08 18:59:05.645717 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:05.645684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v" event={"ID":"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a","Type":"ContainerStarted","Data":"a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3"}
Sep 08 18:59:05.645717 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:05.645721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v" event={"ID":"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a","Type":"ContainerStarted","Data":"0934c557e4f45b7fc2a56a27f1a1a362253e527ab8c2e21208d380a158a17179"}
Sep 08 18:59:05.663008 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:05.662961    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v" podStartSLOduration=1.66294708 podStartE2EDuration="1.66294708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:59:05.661132021 +0000 UTC m=+1506.015225353" watchObservedRunningTime="2026-09-08 18:59:05.66294708 +0000 UTC m=+1506.017040409"
Sep 08 18:59:08.884291 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:08.884255    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:59:14.492635 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:14.492603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:14.493052 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:14.492650    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:14.505088 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:14.505061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:14.699503 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:14.699474    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 18:59:15.579627 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.579598    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"]
Sep 08 18:59:15.583481 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.583445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.585800 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.585782    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3adca991\"/\"openshift-service-ca.crt\""
Sep 08 18:59:15.586297 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.586280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3adca991\"/\"default-dockercfg-28kgp\""
Sep 08 18:59:15.586581 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.586561    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3adca991\"/\"kube-root-ca.crt\""
Sep 08 18:59:15.586626 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.586616    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3adca991\"/\"e2e-lifecycle-model-3adca991-kserve-self-signed-certs\""
Sep 08 18:59:15.590912 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.590887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"]
Sep 08 18:59:15.612374 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.612333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:59:15.672822 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.672790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-model-cache\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.672982 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.672901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqb6j\" (UniqueName: \"kubernetes.io/projected/cf406360-1d2a-4d13-bf89-9c761c2a69ae-kube-api-access-bqb6j\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.672982 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.672927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tls-certs\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.672982 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.672976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-home\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.673092 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.672995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tmp-dir\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.673092 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.673018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-dshm\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.773698 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.773665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-model-cache\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.773857 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.773771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqb6j\" (UniqueName: \"kubernetes.io/projected/cf406360-1d2a-4d13-bf89-9c761c2a69ae-kube-api-access-bqb6j\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.773937 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.773910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tls-certs\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.774014 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.773997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-home\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.774058 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.774038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tmp-dir\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.774149 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.774128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-dshm\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.774206 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.774144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-model-cache\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.774323 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.774302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tmp-dir\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.774473 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.774406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-home\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.776278 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.776260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tls-certs\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.776455 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.776427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-dshm\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.780933 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.780913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqb6j\" (UniqueName: \"kubernetes.io/projected/cf406360-1d2a-4d13-bf89-9c761c2a69ae-kube-api-access-bqb6j\") pod \"e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") " pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:15.894856 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:15.894767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:16.165424 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:16.165400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"]
Sep 08 18:59:16.167468 ip-10-0-140-26 kubenswrapper[2579]: W0908 18:59:16.167442    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf406360_1d2a_4d13_bf89_9c761c2a69ae.slice/crio-6f373dcb0b36b9f07889715f755439091745fd300c6538c940563aaa5bc2be25 WatchSource:0}: Error finding container 6f373dcb0b36b9f07889715f755439091745fd300c6538c940563aaa5bc2be25: Status 404 returned error can't find the container with id 6f373dcb0b36b9f07889715f755439091745fd300c6538c940563aaa5bc2be25
Sep 08 18:59:16.700787 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:16.700749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp" event={"ID":"cf406360-1d2a-4d13-bf89-9c761c2a69ae","Type":"ContainerStarted","Data":"4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b"}
Sep 08 18:59:16.701239 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:16.700794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp" event={"ID":"cf406360-1d2a-4d13-bf89-9c761c2a69ae","Type":"ContainerStarted","Data":"6f373dcb0b36b9f07889715f755439091745fd300c6538c940563aaa5bc2be25"}
Sep 08 18:59:16.718045 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:16.717998    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp" podStartSLOduration=1.7179846570000001 podStartE2EDuration="1.717984657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:59:16.715865391 +0000 UTC m=+1517.069958723" watchObservedRunningTime="2026-09-08 18:59:16.717984657 +0000 UTC m=+1517.072078031"
Sep 08 18:59:22.170771 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:22.170679    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:59:25.895747 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:25.895705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:25.896237 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:25.895760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:25.908441 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:25.908409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:26.764320 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:26.764288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 18:59:36.068232 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:36.068195    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 18:59:46.841823 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:46.841785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 18:59:52.079156 ip-10-0-140-26 kubenswrapper[2579]: I0908 18:59:52.079122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:00:00.166142 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:00.166089    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"]
Sep 08 19:00:00.175532 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:00.172325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"]
Sep 08 19:00:00.175532 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:00.172506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"
Sep 08 19:00:00.179320 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:00.179288    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\""
Sep 08 19:00:00.191221 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:00.191196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"
Sep 08 19:00:00.463403 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:00.463382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"]
Sep 08 19:00:00.465865 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:00:00.465834    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0c3bb1_f810_472b_95d1_4313110d3aff.slice/crio-e6c90644f4b035153c41d041411faea34bec9dc4659331dca264817f57cde712 WatchSource:0}: Error finding container e6c90644f4b035153c41d041411faea34bec9dc4659331dca264817f57cde712: Status 404 returned error can't find the container with id e6c90644f4b035153c41d041411faea34bec9dc4659331dca264817f57cde712
Sep 08 19:00:00.468097 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:00.468077    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 19:00:00.919918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:00.919879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f" event={"ID":"9d0c3bb1-f810-472b-95d1-4313110d3aff","Type":"ContainerStarted","Data":"e6c90644f4b035153c41d041411faea34bec9dc4659331dca264817f57cde712"}
Sep 08 19:00:02.149622 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.149562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:02.185540 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.185504    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"]
Sep 08 19:00:02.186179 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.185745    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5" podUID="398ac735-8b2f-48ce-9c6a-a6b9d637b53b" containerName="payload-processing" containerID="cri-o://a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe" gracePeriod=30
Sep 08 19:00:02.194285 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.194250    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"]
Sep 08 19:00:02.194705 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.194657    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k" podUID="26a29991-18b5-4bed-aa5e-30a8fa23cad0" containerName="payload-pre-processing" containerID="cri-o://3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c" gracePeriod=30
Sep 08 19:00:02.227080 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.227048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:02.298720 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.298682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:02.497870 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.497840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:02.520846 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.520818    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 19:00:02.523875 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.523851    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 19:00:02.632035 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.631996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:02.632283 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.632262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xhzn\" (UniqueName: \"kubernetes.io/projected/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-kube-api-access-6xhzn\") pod \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\" (UID: \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\") "
Sep 08 19:00:02.632363 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.632300    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-plugins-config-volume\") pod \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\" (UID: \"398ac735-8b2f-48ce-9c6a-a6b9d637b53b\") "
Sep 08 19:00:02.632484 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.632468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a29991-18b5-4bed-aa5e-30a8fa23cad0-plugins-config-volume\") pod \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\" (UID: \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\") "
Sep 08 19:00:02.632547 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.632514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gj2d\" (UniqueName: \"kubernetes.io/projected/26a29991-18b5-4bed-aa5e-30a8fa23cad0-kube-api-access-6gj2d\") pod \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\" (UID: \"26a29991-18b5-4bed-aa5e-30a8fa23cad0\") "
Sep 08 19:00:02.632652 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.632635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "398ac735-8b2f-48ce-9c6a-a6b9d637b53b" (UID: "398ac735-8b2f-48ce-9c6a-a6b9d637b53b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:00:02.632878 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.632856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a29991-18b5-4bed-aa5e-30a8fa23cad0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26a29991-18b5-4bed-aa5e-30a8fa23cad0" (UID: "26a29991-18b5-4bed-aa5e-30a8fa23cad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:00:02.632944 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.632913    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:02.634451 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.634421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-kube-api-access-6xhzn" (OuterVolumeSpecName: "kube-api-access-6xhzn") pod "398ac735-8b2f-48ce-9c6a-a6b9d637b53b" (UID: "398ac735-8b2f-48ce-9c6a-a6b9d637b53b"). InnerVolumeSpecName "kube-api-access-6xhzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:00:02.634566 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.634497    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a29991-18b5-4bed-aa5e-30a8fa23cad0-kube-api-access-6gj2d" (OuterVolumeSpecName: "kube-api-access-6gj2d") pod "26a29991-18b5-4bed-aa5e-30a8fa23cad0" (UID: "26a29991-18b5-4bed-aa5e-30a8fa23cad0"). InnerVolumeSpecName "kube-api-access-6gj2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:00:02.734004 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.733921    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xhzn\" (UniqueName: \"kubernetes.io/projected/398ac735-8b2f-48ce-9c6a-a6b9d637b53b-kube-api-access-6xhzn\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:02.734004 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.733949    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a29991-18b5-4bed-aa5e-30a8fa23cad0-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:02.734004 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.733960    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gj2d\" (UniqueName: \"kubernetes.io/projected/26a29991-18b5-4bed-aa5e-30a8fa23cad0-kube-api-access-6gj2d\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:02.931468 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.931430    2579 generic.go:356] "Generic (PLEG): container finished" podID="26a29991-18b5-4bed-aa5e-30a8fa23cad0" containerID="3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c" exitCode=0
Sep 08 19:00:02.931627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.931477    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"
Sep 08 19:00:02.931627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.931519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k" event={"ID":"26a29991-18b5-4bed-aa5e-30a8fa23cad0","Type":"ContainerDied","Data":"3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c"}
Sep 08 19:00:02.931627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.931602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k" event={"ID":"26a29991-18b5-4bed-aa5e-30a8fa23cad0","Type":"ContainerDied","Data":"64a89c3cd73930f83935f1cd5f50ab7d1bc88a110f5a4ed0991e3fc9b4b2c8f6"}
Sep 08 19:00:02.931627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.931628    2579 scope.go:117] "RemoveContainer" containerID="3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c"
Sep 08 19:00:02.932895 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.932873    2579 generic.go:356] "Generic (PLEG): container finished" podID="398ac735-8b2f-48ce-9c6a-a6b9d637b53b" containerID="a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe" exitCode=0
Sep 08 19:00:02.932986 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.932901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5" event={"ID":"398ac735-8b2f-48ce-9c6a-a6b9d637b53b","Type":"ContainerDied","Data":"a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe"}
Sep 08 19:00:02.932986 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.932943    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"
Sep 08 19:00:02.933071 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.932948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5" event={"ID":"398ac735-8b2f-48ce-9c6a-a6b9d637b53b","Type":"ContainerDied","Data":"351d73e19443b25dbffb8fac82dc1348946df64ae1a1dee8aa6504e3cf4f2356"}
Sep 08 19:00:02.941012 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.940805    2579 scope.go:117] "RemoveContainer" containerID="3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c"
Sep 08 19:00:02.941142 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:00:02.941117    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c\": container with ID starting with 3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c not found: ID does not exist" containerID="3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c"
Sep 08 19:00:02.941204 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.941154    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c"} err="failed to get container status \"3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c\": rpc error: code = NotFound desc = could not find container \"3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c\": container with ID starting with 3a85e22956c57161f0e162a0b2324140300170bd8db54bc20c222821b557472c not found: ID does not exist"
Sep 08 19:00:02.941204 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.941176    2579 scope.go:117] "RemoveContainer" containerID="a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe"
Sep 08 19:00:02.949083 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.949068    2579 scope.go:117] "RemoveContainer" containerID="a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe"
Sep 08 19:00:02.949370 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:00:02.949335    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe\": container with ID starting with a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe not found: ID does not exist" containerID="a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe"
Sep 08 19:00:02.949449 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.949379    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe"} err="failed to get container status \"a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe\": rpc error: code = NotFound desc = could not find container \"a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe\": container with ID starting with a7d77195e10b3cc15698b750d7e3cda6308516a4d88cf9314bde132989afafbe not found: ID does not exist"
Sep 08 19:00:02.954764 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.954740    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"]
Sep 08 19:00:02.958229 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.958203    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3adca991-56dcd68b66-rlq7k"]
Sep 08 19:00:02.969075 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.969051    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"]
Sep 08 19:00:02.970809 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:02.970793    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3adca991-778ddd6c5c-28wp5"]
Sep 08 19:00:04.163393 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:04.163358    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a29991-18b5-4bed-aa5e-30a8fa23cad0" path="/var/lib/kubelet/pods/26a29991-18b5-4bed-aa5e-30a8fa23cad0/volumes"
Sep 08 19:00:04.163788 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:04.163772    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398ac735-8b2f-48ce-9c6a-a6b9d637b53b" path="/var/lib/kubelet/pods/398ac735-8b2f-48ce-9c6a-a6b9d637b53b/volumes"
Sep 08 19:00:05.805972 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:05.805943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:08.722679 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:08.722649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:09.322261 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:09.322228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:09.968975 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:09.968939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:00:10.785014 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:10.784978    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"]
Sep 08 19:00:10.881025 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:10.880993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:19.018577 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:19.018535    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d0c3bb1-f810-472b-95d1-4313110d3aff" containerID="369cebc8771e1f5c230801808cce82751b89973b8cdadd359b2d867c546d1ac6" exitCode=0
Sep 08 19:00:19.018930 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:19.018623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f" event={"ID":"9d0c3bb1-f810-472b-95d1-4313110d3aff","Type":"ContainerDied","Data":"369cebc8771e1f5c230801808cce82751b89973b8cdadd359b2d867c546d1ac6"}
Sep 08 19:00:20.156593 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:20.156566    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"
Sep 08 19:00:21.029713 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:21.029684    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"
Sep 08 19:00:21.029889 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:21.029717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f" event={"ID":"9d0c3bb1-f810-472b-95d1-4313110d3aff","Type":"ContainerDied","Data":"e6c90644f4b035153c41d041411faea34bec9dc4659331dca264817f57cde712"}
Sep 08 19:00:21.029889 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:21.029764    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6c90644f4b035153c41d041411faea34bec9dc4659331dca264817f57cde712"
Sep 08 19:00:24.882970 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:24.882933    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:00:25.806685 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:25.806631    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp" podUID="cf406360-1d2a-4d13-bf89-9c761c2a69ae" containerName="main" containerID="cri-o://4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b" gracePeriod=45
Sep 08 19:00:26.054536 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.054507    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 19:00:26.054917 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.054892    2579 generic.go:356] "Generic (PLEG): container finished" podID="cf406360-1d2a-4d13-bf89-9c761c2a69ae" containerID="4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b" exitCode=0
Sep 08 19:00:26.054988 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.054966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp" event={"ID":"cf406360-1d2a-4d13-bf89-9c761c2a69ae","Type":"ContainerDied","Data":"4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b"}
Sep 08 19:00:26.055029 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.055009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp" event={"ID":"cf406360-1d2a-4d13-bf89-9c761c2a69ae","Type":"ContainerDied","Data":"6f373dcb0b36b9f07889715f755439091745fd300c6538c940563aaa5bc2be25"}
Sep 08 19:00:26.055068 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.055026    2579 scope.go:117] "RemoveContainer" containerID="4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b"
Sep 08 19:00:26.063707 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.063695    2579 scope.go:117] "RemoveContainer" containerID="4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b"
Sep 08 19:00:26.063982 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:00:26.063965    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b\": container with ID starting with 4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b not found: ID does not exist" containerID="4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b"
Sep 08 19:00:26.064060 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.063991    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b"} err="failed to get container status \"4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b\": rpc error: code = NotFound desc = could not find container \"4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b\": container with ID starting with 4eb67a89ccc92277c2424dbce9dc48108a50214a70d36b37b445b8079ea40b8b not found: ID does not exist"
Sep 08 19:00:26.162266 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162240    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-home\") pod \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") "
Sep 08 19:00:26.162449 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162284    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tmp-dir\") pod \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") "
Sep 08 19:00:26.162449 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-model-cache\") pod \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") "
Sep 08 19:00:26.162449 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-dshm\") pod \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") "
Sep 08 19:00:26.162449 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tls-certs\") pod \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") "
Sep 08 19:00:26.162678 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqb6j\" (UniqueName: \"kubernetes.io/projected/cf406360-1d2a-4d13-bf89-9c761c2a69ae-kube-api-access-bqb6j\") pod \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\" (UID: \"cf406360-1d2a-4d13-bf89-9c761c2a69ae\") "
Sep 08 19:00:26.162678 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-home" (OuterVolumeSpecName: "home") pod "cf406360-1d2a-4d13-bf89-9c761c2a69ae" (UID: "cf406360-1d2a-4d13-bf89-9c761c2a69ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:00:26.162678 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf406360-1d2a-4d13-bf89-9c761c2a69ae" (UID: "cf406360-1d2a-4d13-bf89-9c761c2a69ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:00:26.162678 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf406360-1d2a-4d13-bf89-9c761c2a69ae" (UID: "cf406360-1d2a-4d13-bf89-9c761c2a69ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:00:26.162883 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162810    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:26.162883 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162830    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:26.162883 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.162843    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:26.164703 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.164674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-dshm" (OuterVolumeSpecName: "dshm") pod "cf406360-1d2a-4d13-bf89-9c761c2a69ae" (UID: "cf406360-1d2a-4d13-bf89-9c761c2a69ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:00:26.164849 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.164737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf406360-1d2a-4d13-bf89-9c761c2a69ae-kube-api-access-bqb6j" (OuterVolumeSpecName: "kube-api-access-bqb6j") pod "cf406360-1d2a-4d13-bf89-9c761c2a69ae" (UID: "cf406360-1d2a-4d13-bf89-9c761c2a69ae"). InnerVolumeSpecName "kube-api-access-bqb6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:00:26.164849 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.164795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf406360-1d2a-4d13-bf89-9c761c2a69ae" (UID: "cf406360-1d2a-4d13-bf89-9c761c2a69ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:00:26.263500 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.263460    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf406360-1d2a-4d13-bf89-9c761c2a69ae-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:26.263500 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.263494    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf406360-1d2a-4d13-bf89-9c761c2a69ae-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:26.263500 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:26.263509    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqb6j\" (UniqueName: \"kubernetes.io/projected/cf406360-1d2a-4d13-bf89-9c761c2a69ae-kube-api-access-bqb6j\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:00:27.061220 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:27.061186    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"
Sep 08 19:00:27.081557 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:27.081531    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"]
Sep 08 19:00:27.084983 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:27.084963    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3adca991/e2e-lifecycle-model-3adca991-kserve-95b76d9ff-tfcjp"]
Sep 08 19:00:28.162715 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:28.162677    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf406360-1d2a-4d13-bf89-9c761c2a69ae" path="/var/lib/kubelet/pods/cf406360-1d2a-4d13-bf89-9c761c2a69ae/volumes"
Sep 08 19:00:40.284675 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:40.284640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:00:47.656626 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:47.656589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:47.882467 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:47.882435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:53.595527 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:53.595493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:00:55.371005 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:00:55.370967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:01:07.786899 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:07.786859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:07.887451 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:07.887419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:08.080610 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:08.080576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:09.431299 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:09.431266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:09.527235 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:09.527206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:09.628160 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:09.628130    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:09.966569 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:09.966539    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"]
Sep 08 19:01:09.977871 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:09.977840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:10.298018 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:10.297931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:10.391597 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:10.391563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:10.496789 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:10.496761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:10.901670 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:10.901634    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"]
Sep 08 19:01:10.910504 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:10.910476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:11.481643 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:11.481601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:01:11.861687 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:11.861658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:17.840466 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:17.840429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:17.903740 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:17.903709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:18.592853 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:18.592820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:21.584883 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.584849    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"]
Sep 08 19:01:21.585490 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585469    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf406360-1d2a-4d13-bf89-9c761c2a69ae" containerName="main"
Sep 08 19:01:21.585546 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585494    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf406360-1d2a-4d13-bf89-9c761c2a69ae" containerName="main"
Sep 08 19:01:21.585546 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585507    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0c3bb1-f810-472b-95d1-4313110d3aff" containerName="cleanup"
Sep 08 19:01:21.585546 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585516    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0c3bb1-f810-472b-95d1-4313110d3aff" containerName="cleanup"
Sep 08 19:01:21.585546 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585544    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a29991-18b5-4bed-aa5e-30a8fa23cad0" containerName="payload-pre-processing"
Sep 08 19:01:21.585684 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585554    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a29991-18b5-4bed-aa5e-30a8fa23cad0" containerName="payload-pre-processing"
Sep 08 19:01:21.585684 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585582    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398ac735-8b2f-48ce-9c6a-a6b9d637b53b" containerName="payload-processing"
Sep 08 19:01:21.585684 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585591    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="398ac735-8b2f-48ce-9c6a-a6b9d637b53b" containerName="payload-processing"
Sep 08 19:01:21.585789 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585691    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a29991-18b5-4bed-aa5e-30a8fa23cad0" containerName="payload-pre-processing"
Sep 08 19:01:21.585789 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585701    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0c3bb1-f810-472b-95d1-4313110d3aff" containerName="cleanup"
Sep 08 19:01:21.585789 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585713    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf406360-1d2a-4d13-bf89-9c761c2a69ae" containerName="main"
Sep 08 19:01:21.585789 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.585726    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="398ac735-8b2f-48ce-9c6a-a6b9d637b53b" containerName="payload-processing"
Sep 08 19:01:21.589316 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.589299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.591369 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.591329    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a4001b\"/\"sub-test-model-a4001b-kserve-self-signed-certs\""
Sep 08 19:01:21.594938 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.594917    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"]
Sep 08 19:01:21.605499 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.605477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:21.700193 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.700160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-dshm\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.700374 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.700211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zfq\" (UniqueName: \"kubernetes.io/projected/0e6a569c-994b-4767-9426-9c1ba886cabe-kube-api-access-75zfq\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.700374 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.700275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6a569c-994b-4767-9426-9c1ba886cabe-tls-certs\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.700374 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.700318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-home\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.700374 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.700336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-model-cache\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.700540 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.700436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-tmp-dir\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801079 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-dshm\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zfq\" (UniqueName: \"kubernetes.io/projected/0e6a569c-994b-4767-9426-9c1ba886cabe-kube-api-access-75zfq\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6a569c-994b-4767-9426-9c1ba886cabe-tls-certs\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-home\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-model-cache\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-tmp-dir\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801612 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-home\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801694 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-tmp-dir\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.801739 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.801680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-model-cache\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.803498 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.803479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-dshm\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.803782 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.803765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6a569c-994b-4767-9426-9c1ba886cabe-tls-certs\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.809180 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.809154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zfq\" (UniqueName: \"kubernetes.io/projected/0e6a569c-994b-4767-9426-9c1ba886cabe-kube-api-access-75zfq\") pod \"sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:21.899863 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:21.899779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:22.171332 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:22.171305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"]
Sep 08 19:01:22.173251 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:01:22.173226    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e6a569c_994b_4767_9426_9c1ba886cabe.slice/crio-8e9b3654f9129253073bc4dbecb692f1fc729ce7a1a1a34dabc108fc14ef6ed7 WatchSource:0}: Error finding container 8e9b3654f9129253073bc4dbecb692f1fc729ce7a1a1a34dabc108fc14ef6ed7: Status 404 returned error can't find the container with id 8e9b3654f9129253073bc4dbecb692f1fc729ce7a1a1a34dabc108fc14ef6ed7
Sep 08 19:01:22.340612 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:22.340579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8" event={"ID":"0e6a569c-994b-4767-9426-9c1ba886cabe","Type":"ContainerStarted","Data":"f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6"}
Sep 08 19:01:22.340612 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:22.340621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8" event={"ID":"0e6a569c-994b-4767-9426-9c1ba886cabe","Type":"ContainerStarted","Data":"8e9b3654f9129253073bc4dbecb692f1fc729ce7a1a1a34dabc108fc14ef6ed7"}
Sep 08 19:01:22.356867 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:22.356816    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8" podStartSLOduration=1.356801102 podStartE2EDuration="1.356801102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:01:22.355367942 +0000 UTC m=+1642.709461271" watchObservedRunningTime="2026-09-08 19:01:22.356801102 +0000 UTC m=+1642.710894435"
Sep 08 19:01:24.751685 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:24.751638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:24.857523 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:24.857494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:24.958422 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:24.958382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:24.987634 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:24.987598    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l" podUID="5621170d-06d6-44a8-bad0-72015e2ae9d8" containerName="main" containerID="cri-o://1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0" gracePeriod=45
Sep 08 19:01:25.092323 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.092294    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:01:25.239437 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.239413    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 19:01:25.337236 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337208    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj8td\" (UniqueName: \"kubernetes.io/projected/5621170d-06d6-44a8-bad0-72015e2ae9d8-kube-api-access-vj8td\") pod \"5621170d-06d6-44a8-bad0-72015e2ae9d8\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") "
Sep 08 19:01:25.337494 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5621170d-06d6-44a8-bad0-72015e2ae9d8-tls-certs\") pod \"5621170d-06d6-44a8-bad0-72015e2ae9d8\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") "
Sep 08 19:01:25.337494 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-dshm\") pod \"5621170d-06d6-44a8-bad0-72015e2ae9d8\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") "
Sep 08 19:01:25.337494 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-model-cache\") pod \"5621170d-06d6-44a8-bad0-72015e2ae9d8\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") "
Sep 08 19:01:25.337494 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-home\") pod \"5621170d-06d6-44a8-bad0-72015e2ae9d8\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") "
Sep 08 19:01:25.337494 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-tmp-dir\") pod \"5621170d-06d6-44a8-bad0-72015e2ae9d8\" (UID: \"5621170d-06d6-44a8-bad0-72015e2ae9d8\") "
Sep 08 19:01:25.337775 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337671    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "5621170d-06d6-44a8-bad0-72015e2ae9d8" (UID: "5621170d-06d6-44a8-bad0-72015e2ae9d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:01:25.337775 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-home" (OuterVolumeSpecName: "home") pod "5621170d-06d6-44a8-bad0-72015e2ae9d8" (UID: "5621170d-06d6-44a8-bad0-72015e2ae9d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:01:25.337775 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5621170d-06d6-44a8-bad0-72015e2ae9d8" (UID: "5621170d-06d6-44a8-bad0-72015e2ae9d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:01:25.337897 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337840    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:25.337897 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337852    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:25.337897 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.337860    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:25.339538 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.339518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-dshm" (OuterVolumeSpecName: "dshm") pod "5621170d-06d6-44a8-bad0-72015e2ae9d8" (UID: "5621170d-06d6-44a8-bad0-72015e2ae9d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:01:25.339836 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.339816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5621170d-06d6-44a8-bad0-72015e2ae9d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5621170d-06d6-44a8-bad0-72015e2ae9d8" (UID: "5621170d-06d6-44a8-bad0-72015e2ae9d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:01:25.339836 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.339821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5621170d-06d6-44a8-bad0-72015e2ae9d8-kube-api-access-vj8td" (OuterVolumeSpecName: "kube-api-access-vj8td") pod "5621170d-06d6-44a8-bad0-72015e2ae9d8" (UID: "5621170d-06d6-44a8-bad0-72015e2ae9d8"). InnerVolumeSpecName "kube-api-access-vj8td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:01:25.358647 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.358621    2579 generic.go:356] "Generic (PLEG): container finished" podID="5621170d-06d6-44a8-bad0-72015e2ae9d8" containerID="1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0" exitCode=0
Sep 08 19:01:25.358756 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.358665    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"
Sep 08 19:01:25.358756 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.358663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l" event={"ID":"5621170d-06d6-44a8-bad0-72015e2ae9d8","Type":"ContainerDied","Data":"1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0"}
Sep 08 19:01:25.358864 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.358775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l" event={"ID":"5621170d-06d6-44a8-bad0-72015e2ae9d8","Type":"ContainerDied","Data":"a705585b09a2f2f09de1088b9024b539149f6099a937bc39baccb633a91ef7c4"}
Sep 08 19:01:25.358864 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.358793    2579 scope.go:117] "RemoveContainer" containerID="1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0"
Sep 08 19:01:25.367425 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.367374    2579 scope.go:117] "RemoveContainer" containerID="1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0"
Sep 08 19:01:25.367668 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:01:25.367651    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0\": container with ID starting with 1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0 not found: ID does not exist" containerID="1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0"
Sep 08 19:01:25.367724 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.367677    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0"} err="failed to get container status \"1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0\": rpc error: code = NotFound desc = could not find container \"1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0\": container with ID starting with 1f1b74f8b64124cfd28eb58a6fe38aaf406508e8fcbeec287875f07a91d279c0 not found: ID does not exist"
Sep 08 19:01:25.379139 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.379114    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"]
Sep 08 19:01:25.390481 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.390460    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/auth-test-model-a4001b-kserve-cb5568475-n685l"]
Sep 08 19:01:25.439236 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.439209    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5621170d-06d6-44a8-bad0-72015e2ae9d8-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:25.439236 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.439231    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8td\" (UniqueName: \"kubernetes.io/projected/5621170d-06d6-44a8-bad0-72015e2ae9d8-kube-api-access-vj8td\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:25.439392 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.439241    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5621170d-06d6-44a8-bad0-72015e2ae9d8-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:25.921292 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:25.921251    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v" podUID="cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" containerName="main" containerID="cri-o://a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3" gracePeriod=45
Sep 08 19:01:26.164585 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.164555    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5621170d-06d6-44a8-bad0-72015e2ae9d8" path="/var/lib/kubelet/pods/5621170d-06d6-44a8-bad0-72015e2ae9d8/volumes"
Sep 08 19:01:26.176981 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.176928    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 19:01:26.247358 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-model-cache\") pod \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") "
Sep 08 19:01:26.247543 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tmp-dir\") pod \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") "
Sep 08 19:01:26.247543 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247415    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tls-certs\") pod \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") "
Sep 08 19:01:26.247543 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v4lc\" (UniqueName: \"kubernetes.io/projected/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-kube-api-access-7v4lc\") pod \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") "
Sep 08 19:01:26.247543 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-dshm\") pod \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") "
Sep 08 19:01:26.247755 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-home\") pod \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\" (UID: \"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a\") "
Sep 08 19:01:26.247755 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-model-cache" (OuterVolumeSpecName: "model-cache") pod "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" (UID: "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:01:26.247755 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247622    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" (UID: "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:01:26.247898 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-home" (OuterVolumeSpecName: "home") pod "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" (UID: "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:01:26.247948 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247926    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:26.247948 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247939    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:26.247948 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.247947    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:26.249817 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.249781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" (UID: "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:01:26.249933 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.249850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-dshm" (OuterVolumeSpecName: "dshm") pod "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" (UID: "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:01:26.249933 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.249881    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-kube-api-access-7v4lc" (OuterVolumeSpecName: "kube-api-access-7v4lc") pod "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" (UID: "cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a"). InnerVolumeSpecName "kube-api-access-7v4lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:01:26.348489 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.348450    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:26.348489 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.348484    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v4lc\" (UniqueName: \"kubernetes.io/projected/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-kube-api-access-7v4lc\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:26.348489 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.348494    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:01:26.366557 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.366524    2579 generic.go:356] "Generic (PLEG): container finished" podID="cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" containerID="a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3" exitCode=0
Sep 08 19:01:26.366726 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.366567    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"
Sep 08 19:01:26.366726 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.366611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v" event={"ID":"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a","Type":"ContainerDied","Data":"a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3"}
Sep 08 19:01:26.366726 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.366657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v" event={"ID":"cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a","Type":"ContainerDied","Data":"0934c557e4f45b7fc2a56a27f1a1a362253e527ab8c2e21208d380a158a17179"}
Sep 08 19:01:26.366726 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.366678    2579 scope.go:117] "RemoveContainer" containerID="a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3"
Sep 08 19:01:26.377993 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.377968    2579 scope.go:117] "RemoveContainer" containerID="a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3"
Sep 08 19:01:26.378366 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:01:26.378326    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3\": container with ID starting with a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3 not found: ID does not exist" containerID="a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3"
Sep 08 19:01:26.378434 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.378377    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3"} err="failed to get container status \"a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3\": rpc error: code = NotFound desc = could not find container \"a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3\": container with ID starting with a9efca3a87c3eeb79a9baa3f82cb14e8ed971118dfc664789dd480c727a702f3 not found: ID does not exist"
Sep 08 19:01:26.393775 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.393749    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"]
Sep 08 19:01:26.397627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:26.397608    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/auth-test-model-525196-kserve-749fb46699-kqv6v"]
Sep 08 19:01:28.164206 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:28.164171    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" path="/var/lib/kubelet/pods/cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a/volumes"
Sep 08 19:01:29.586824 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:29.586794    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:31.900984 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:31.900954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:31.901514 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:31.900997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:31.913739 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:31.913714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:32.408139 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:32.408112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:01:35.797741 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:35.797698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:35.975240 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:35.975202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:44.790519 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.790478    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"]
Sep 08 19:01:44.790916 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.790879    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" containerName="main"
Sep 08 19:01:44.790916 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.790891    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" containerName="main"
Sep 08 19:01:44.790916 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.790910    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5621170d-06d6-44a8-bad0-72015e2ae9d8" containerName="main"
Sep 08 19:01:44.790916 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.790915    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5621170d-06d6-44a8-bad0-72015e2ae9d8" containerName="main"
Sep 08 19:01:44.791043 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.790970    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea4e9e3-ee9e-43a8-8a90-7f3af28ba87a" containerName="main"
Sep 08 19:01:44.791043 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.790980    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5621170d-06d6-44a8-bad0-72015e2ae9d8" containerName="main"
Sep 08 19:01:44.794180 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.794159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.797042 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.797020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"]
Sep 08 19:01:44.797167 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.797127    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"default-dockercfg-tbv89\""
Sep 08 19:01:44.797245 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.797158    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"openshift-service-ca.crt\""
Sep 08 19:01:44.797245 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.797167    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"kube-root-ca.crt\""
Sep 08 19:01:44.797369 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.797277    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"sub-test-model-525196-kserve-self-signed-certs\""
Sep 08 19:01:44.824446 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.824413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tls-certs\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.824561 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.824465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-dshm\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.824561 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.824552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tmp-dir\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.824647 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.824576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-home\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.824647 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.824623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-model-cache\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.824647 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.824643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdslp\" (UniqueName: \"kubernetes.io/projected/fcfcf20d-ce79-4a37-8d75-355de6f26f80-kube-api-access-rdslp\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.899913 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.899883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:01:44.926093 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-model-cache\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.926253 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdslp\" (UniqueName: \"kubernetes.io/projected/fcfcf20d-ce79-4a37-8d75-355de6f26f80-kube-api-access-rdslp\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.926253 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tls-certs\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.926253 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-dshm\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.926253 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tmp-dir\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.926253 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-home\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.926568 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-model-cache\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.926644 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-home\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.926716 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.926691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tmp-dir\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.928516 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.928494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-dshm\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.928737 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.928720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tls-certs\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:44.933877 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:44.933856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdslp\" (UniqueName: \"kubernetes.io/projected/fcfcf20d-ce79-4a37-8d75-355de6f26f80-kube-api-access-rdslp\") pod \"sub-test-model-525196-kserve-57595bf8dc-m5zcx\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") " pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:45.104714 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:45.104681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:45.374612 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:45.374587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"]
Sep 08 19:01:45.376715 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:01:45.376674    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcfcf20d_ce79_4a37_8d75_355de6f26f80.slice/crio-ec4165d8cde91f9aca8e05988ced9f7e563f03c115fe44034437dd4fbe7e9d4b WatchSource:0}: Error finding container ec4165d8cde91f9aca8e05988ced9f7e563f03c115fe44034437dd4fbe7e9d4b: Status 404 returned error can't find the container with id ec4165d8cde91f9aca8e05988ced9f7e563f03c115fe44034437dd4fbe7e9d4b
Sep 08 19:01:45.463945 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:45.463912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx" event={"ID":"fcfcf20d-ce79-4a37-8d75-355de6f26f80","Type":"ContainerStarted","Data":"855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900"}
Sep 08 19:01:45.463945 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:45.463952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx" event={"ID":"fcfcf20d-ce79-4a37-8d75-355de6f26f80","Type":"ContainerStarted","Data":"ec4165d8cde91f9aca8e05988ced9f7e563f03c115fe44034437dd4fbe7e9d4b"}
Sep 08 19:01:45.481152 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:45.481104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx" podStartSLOduration=1.481088414 podStartE2EDuration="1.481088414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:01:45.479959303 +0000 UTC m=+1665.834052636" watchObservedRunningTime="2026-09-08 19:01:45.481088414 +0000 UTC m=+1665.835181746"
Sep 08 19:01:49.376827 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:49.376792    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:01:55.105404 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:55.105328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:55.105404 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:55.105417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:55.118121 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:55.118091    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:55.525946 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:55.525863    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:01:56.785803 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:01:56.785769    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:02.669880 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:02.669844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:02.755238 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:02.755205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:03.041572 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:03.041493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:03.361975 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:03.361921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:03.611842 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:03.611809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:06.990432 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:06.990392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:07.090590 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:07.090557    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:08.231475 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:08.231426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:08.326313 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:08.326278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:08.426136 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:08.426102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:08.914267 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:08.914231    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"]
Sep 08 19:02:08.917809 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:08.917792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:08.919838 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:08.919818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-965746-dockercfg-cnb68\""
Sep 08 19:02:08.920097 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:08.920081    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-965746\""
Sep 08 19:02:08.928607 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:08.928583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"]
Sep 08 19:02:09.032438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.032408    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:09.069324 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.069284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11e275-98f8-4ec0-a398-ddc55d23c630-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs\" (UID: \"bc11e275-98f8-4ec0-a398-ddc55d23c630\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:09.069519 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.069369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpvm\" (UniqueName: \"kubernetes.io/projected/bc11e275-98f8-4ec0-a398-ddc55d23c630-kube-api-access-xfpvm\") pod \"payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs\" (UID: \"bc11e275-98f8-4ec0-a398-ddc55d23c630\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:09.161844 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.161813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:09.170126 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.170055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11e275-98f8-4ec0-a398-ddc55d23c630-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs\" (UID: \"bc11e275-98f8-4ec0-a398-ddc55d23c630\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:09.170126 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.170111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpvm\" (UniqueName: \"kubernetes.io/projected/bc11e275-98f8-4ec0-a398-ddc55d23c630-kube-api-access-xfpvm\") pod \"payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs\" (UID: \"bc11e275-98f8-4ec0-a398-ddc55d23c630\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:09.170720 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.170703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11e275-98f8-4ec0-a398-ddc55d23c630-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs\" (UID: \"bc11e275-98f8-4ec0-a398-ddc55d23c630\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:09.178294 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.178276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpvm\" (UniqueName: \"kubernetes.io/projected/bc11e275-98f8-4ec0-a398-ddc55d23c630-kube-api-access-xfpvm\") pod \"payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs\" (UID: \"bc11e275-98f8-4ec0-a398-ddc55d23c630\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:09.228462 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.228425    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:09.502037 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.502009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"]
Sep 08 19:02:09.504231 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:02:09.504197    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc11e275_98f8_4ec0_a398_ddc55d23c630.slice/crio-7639ef141216d387fe5b94fe273d5384a0da067d9b57671cf741b59146ca0f09 WatchSource:0}: Error finding container 7639ef141216d387fe5b94fe273d5384a0da067d9b57671cf741b59146ca0f09: Status 404 returned error can't find the container with id 7639ef141216d387fe5b94fe273d5384a0da067d9b57671cf741b59146ca0f09
Sep 08 19:02:09.572352 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.572314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:02:09.583708 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.583683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs" event={"ID":"bc11e275-98f8-4ec0-a398-ddc55d23c630","Type":"ContainerStarted","Data":"d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4"}
Sep 08 19:02:09.583816 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.583727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs" event={"ID":"bc11e275-98f8-4ec0-a398-ddc55d23c630","Type":"ContainerStarted","Data":"7639ef141216d387fe5b94fe273d5384a0da067d9b57671cf741b59146ca0f09"}
Sep 08 19:02:09.583875 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.583832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:09.599668 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:09.599623    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs" podStartSLOduration=1.59960781 podStartE2EDuration="1.59960781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:09.598286973 +0000 UTC m=+1689.952380300" watchObservedRunningTime="2026-09-08 19:02:09.59960781 +0000 UTC m=+1689.953701178"
Sep 08 19:02:20.591494 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:20.591464    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:02:23.152978 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.152905    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:23.402732 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.402698    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-3712fc"]
Sep 08 19:02:23.407412 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.407332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:23.410136 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.410115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgjvd\""
Sep 08 19:02:23.422547 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.422522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-3712fc"]
Sep 08 19:02:23.508739 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.508705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8p2h\" (UniqueName: \"kubernetes.io/projected/4c99f219-83d4-4273-b891-29d1de6485df-kube-api-access-k8p2h\") pod \"test-curl-7695-3712fc\" (UID: \"4c99f219-83d4-4273-b891-29d1de6485df\") " pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:23.609721 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.609678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8p2h\" (UniqueName: \"kubernetes.io/projected/4c99f219-83d4-4273-b891-29d1de6485df-kube-api-access-k8p2h\") pod \"test-curl-7695-3712fc\" (UID: \"4c99f219-83d4-4273-b891-29d1de6485df\") " pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:23.618090 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.618054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8p2h\" (UniqueName: \"kubernetes.io/projected/4c99f219-83d4-4273-b891-29d1de6485df-kube-api-access-k8p2h\") pod \"test-curl-7695-3712fc\" (UID: \"4c99f219-83d4-4273-b891-29d1de6485df\") " pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:23.716661 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.716564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:23.985139 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:23.985114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-3712fc"]
Sep 08 19:02:23.987222 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:02:23.987198    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c99f219_83d4_4273_b891_29d1de6485df.slice/crio-8c5cfa8d8263afd0b619ff8ec196456036da9d40421d0e761de52b77d52130ca WatchSource:0}: Error finding container 8c5cfa8d8263afd0b619ff8ec196456036da9d40421d0e761de52b77d52130ca: Status 404 returned error can't find the container with id 8c5cfa8d8263afd0b619ff8ec196456036da9d40421d0e761de52b77d52130ca
Sep 08 19:02:24.662536 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:24.662500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-3712fc" event={"ID":"4c99f219-83d4-4273-b891-29d1de6485df","Type":"ContainerStarted","Data":"8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075"}
Sep 08 19:02:24.662536 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:24.662537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-3712fc" event={"ID":"4c99f219-83d4-4273-b891-29d1de6485df","Type":"ContainerStarted","Data":"8c5cfa8d8263afd0b619ff8ec196456036da9d40421d0e761de52b77d52130ca"}
Sep 08 19:02:24.677503 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:24.677455    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-3712fc" podStartSLOduration=1.6774207799999998 podStartE2EDuration="1.67742078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:24.676922418 +0000 UTC m=+1705.031015751" watchObservedRunningTime="2026-09-08 19:02:24.67742078 +0000 UTC m=+1705.031514160"
Sep 08 19:02:25.310572 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:25.310541    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-3712fc"]
Sep 08 19:02:25.313803 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:25.313778    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-3712fc"]
Sep 08 19:02:25.667946 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:02:25.667915    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-3712fc" container="test-curl-7695-3712fc"
Sep 08 19:02:26.673385 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:26.673309    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-3712fc" podUID="4c99f219-83d4-4273-b891-29d1de6485df" containerName="test-curl-7695-3712fc" containerID="cri-o://8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075" gracePeriod=2
Sep 08 19:02:28.782996 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:28.782969    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:28.862291 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:28.862259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8p2h\" (UniqueName: \"kubernetes.io/projected/4c99f219-83d4-4273-b891-29d1de6485df-kube-api-access-k8p2h\") pod \"4c99f219-83d4-4273-b891-29d1de6485df\" (UID: \"4c99f219-83d4-4273-b891-29d1de6485df\") "
Sep 08 19:02:28.864326 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:28.864298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c99f219-83d4-4273-b891-29d1de6485df-kube-api-access-k8p2h" (OuterVolumeSpecName: "kube-api-access-k8p2h") pod "4c99f219-83d4-4273-b891-29d1de6485df" (UID: "4c99f219-83d4-4273-b891-29d1de6485df"). InnerVolumeSpecName "kube-api-access-k8p2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:02:28.963790 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:28.963709    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8p2h\" (UniqueName: \"kubernetes.io/projected/4c99f219-83d4-4273-b891-29d1de6485df-kube-api-access-k8p2h\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:29.134226 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:29.134188    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:29.693688 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:29.693436    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c99f219-83d4-4273-b891-29d1de6485df" containerID="8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075" exitCode=137
Sep 08 19:02:29.693688 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:29.693487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:29.693688 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:29.693533    2579 scope.go:117] "RemoveContainer" containerID="8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075"
Sep 08 19:02:29.709883 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:02:29.709840    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-3712fc\" is forbidden: User \"system:node:ip-10-0-140-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-26.ec2.internal' and this object" podUID="4c99f219-83d4-4273-b891-29d1de6485df" pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:29.710974 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:29.710946    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"]
Sep 08 19:02:29.713962 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:29.713942    2579 scope.go:117] "RemoveContainer" containerID="8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075"
Sep 08 19:02:29.714331 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:02:29.714310    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075\": container with ID starting with 8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075 not found: ID does not exist" containerID="8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075"
Sep 08 19:02:29.714419 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:29.714357    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075"} err="failed to get container status \"8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075\": rpc error: code = NotFound desc = could not find container \"8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075\": container with ID starting with 8cf3808838867761245deef03a8e87c0d7d45f4f2f254bcba79d8d1e3191a075 not found: ID does not exist"
Sep 08 19:02:29.730264 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:02:29.730227    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-3712fc\" is forbidden: User \"system:node:ip-10-0-140-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-26.ec2.internal' and this object" podUID="4c99f219-83d4-4273-b891-29d1de6485df" pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:29.737184 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:29.737163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:30.161429 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:02:30.161391    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-3712fc\" is forbidden: User \"system:node:ip-10-0-140-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-26.ec2.internal' and this object" podUID="4c99f219-83d4-4273-b891-29d1de6485df" pod="opendatahub/test-curl-7695-3712fc"
Sep 08 19:02:30.163863 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:30.163841    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c99f219-83d4-4273-b891-29d1de6485df" path="/var/lib/kubelet/pods/4c99f219-83d4-4273-b891-29d1de6485df/volumes"
Sep 08 19:02:30.781870 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:30.781830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"]
Sep 08 19:02:30.802280 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:30.802245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:31.945301 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:31.945273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:32.381407 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:32.381371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:32.684660 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:32.684571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:33.332531 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:33.332503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:35.487236 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:35.487198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:02:36.001950 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:36.001916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:36.099279 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:36.099241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:37.130454 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.130414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"]
Sep 08 19:02:37.130843 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.130822    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c99f219-83d4-4273-b891-29d1de6485df" containerName="test-curl-7695-3712fc"
Sep 08 19:02:37.130843 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.130832    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c99f219-83d4-4273-b891-29d1de6485df" containerName="test-curl-7695-3712fc"
Sep 08 19:02:37.130927 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.130917    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c99f219-83d4-4273-b891-29d1de6485df" containerName="test-curl-7695-3712fc"
Sep 08 19:02:37.134131 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.134115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.136503 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.136481    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ee8372-serving-cert\""
Sep 08 19:02:37.147717 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.147696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"]
Sep 08 19:02:37.179559 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.179530    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:37.251119 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.251089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-metrics-certs\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.251359 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.251137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-openshift-service-ca\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.251359 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.251214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-maas-api-tls\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.251359 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.251274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmmck\" (UniqueName: \"kubernetes.io/projected/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-kube-api-access-kmmck\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.278850 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.278813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:37.352626 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.352588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-openshift-service-ca\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.352793 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.352639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-maas-api-tls\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.352793 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.352688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmmck\" (UniqueName: \"kubernetes.io/projected/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-kube-api-access-kmmck\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.352793 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.352753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-metrics-certs\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.353325 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.353300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-openshift-service-ca\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.354950 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.354925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-maas-api-tls\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.355050 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.355023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-metrics-certs\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.360555 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.360535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmmck\" (UniqueName: \"kubernetes.io/projected/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-kube-api-access-kmmck\") pod \"maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.445255 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.445172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:37.641836 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.641802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:37.719949 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.719926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"]
Sep 08 19:02:37.721854 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.721823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:37.722474 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:02:37.722450    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4b1a7f_ef57_42fe_9b90_7dda15b15184.slice/crio-8fc7b0bb510e0f8b83cc1ae115ca9349c0773d560f62070215660dcbbdc16e80 WatchSource:0}: Error finding container 8fc7b0bb510e0f8b83cc1ae115ca9349c0773d560f62070215660dcbbdc16e80: Status 404 returned error can't find the container with id 8fc7b0bb510e0f8b83cc1ae115ca9349c0773d560f62070215660dcbbdc16e80
Sep 08 19:02:37.734352 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:37.734313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99" event={"ID":"1e4b1a7f-ef57-42fe-9b90-7dda15b15184","Type":"ContainerStarted","Data":"8fc7b0bb510e0f8b83cc1ae115ca9349c0773d560f62070215660dcbbdc16e80"}
Sep 08 19:02:38.741055 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:38.741022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99" event={"ID":"1e4b1a7f-ef57-42fe-9b90-7dda15b15184","Type":"ContainerStarted","Data":"476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4"}
Sep 08 19:02:38.741520 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:38.741128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:38.758271 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:38.758224    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99" podStartSLOduration=1.293843388 podStartE2EDuration="1.75820992s" totalImagesPullingTime="464.366532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:38.756685252 +0000 UTC m=+1719.110778583" watchObservedRunningTime="2026-09-08 19:02:38.75820992 +0000 UTC m=+1719.112303253"
Sep 08 19:02:41.981415 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:41.981383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:44.730496 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:44.730453    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8" podUID="0e6a569c-994b-4767-9426-9c1ba886cabe" containerName="main" containerID="cri-o://f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6" gracePeriod=45
Sep 08 19:02:44.751894 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:44.751871    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:02:44.975264 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:44.975236    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:02:45.128938 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.128906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6a569c-994b-4767-9426-9c1ba886cabe-tls-certs\") pod \"0e6a569c-994b-4767-9426-9c1ba886cabe\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") "
Sep 08 19:02:45.129098 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.128986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-tmp-dir\") pod \"0e6a569c-994b-4767-9426-9c1ba886cabe\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") "
Sep 08 19:02:45.129139 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129092    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-model-cache\") pod \"0e6a569c-994b-4767-9426-9c1ba886cabe\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") "
Sep 08 19:02:45.129139 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75zfq\" (UniqueName: \"kubernetes.io/projected/0e6a569c-994b-4767-9426-9c1ba886cabe-kube-api-access-75zfq\") pod \"0e6a569c-994b-4767-9426-9c1ba886cabe\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") "
Sep 08 19:02:45.129248 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129156    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-dshm\") pod \"0e6a569c-994b-4767-9426-9c1ba886cabe\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") "
Sep 08 19:02:45.129248 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-home\") pod \"0e6a569c-994b-4767-9426-9c1ba886cabe\" (UID: \"0e6a569c-994b-4767-9426-9c1ba886cabe\") "
Sep 08 19:02:45.129386 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129243    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e6a569c-994b-4767-9426-9c1ba886cabe" (UID: "0e6a569c-994b-4767-9426-9c1ba886cabe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:02:45.129600 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129404    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e6a569c-994b-4767-9426-9c1ba886cabe" (UID: "0e6a569c-994b-4767-9426-9c1ba886cabe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:02:45.129600 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129553    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-home" (OuterVolumeSpecName: "home") pod "0e6a569c-994b-4767-9426-9c1ba886cabe" (UID: "0e6a569c-994b-4767-9426-9c1ba886cabe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:02:45.129722 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129664    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:45.129722 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.129683    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:45.131106 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.131086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6a569c-994b-4767-9426-9c1ba886cabe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e6a569c-994b-4767-9426-9c1ba886cabe" (UID: "0e6a569c-994b-4767-9426-9c1ba886cabe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:02:45.131232 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.131211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-dshm" (OuterVolumeSpecName: "dshm") pod "0e6a569c-994b-4767-9426-9c1ba886cabe" (UID: "0e6a569c-994b-4767-9426-9c1ba886cabe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:02:45.131364 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.131328    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6a569c-994b-4767-9426-9c1ba886cabe-kube-api-access-75zfq" (OuterVolumeSpecName: "kube-api-access-75zfq") pod "0e6a569c-994b-4767-9426-9c1ba886cabe" (UID: "0e6a569c-994b-4767-9426-9c1ba886cabe"). InnerVolumeSpecName "kube-api-access-75zfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:02:45.231036 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.231003    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zfq\" (UniqueName: \"kubernetes.io/projected/0e6a569c-994b-4767-9426-9c1ba886cabe-kube-api-access-75zfq\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:45.231036 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.231029    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:45.231036 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.231038    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6a569c-994b-4767-9426-9c1ba886cabe-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:45.231265 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.231045    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6a569c-994b-4767-9426-9c1ba886cabe-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:45.370974 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.370940    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"]
Sep 08 19:02:45.371433 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.371410    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6a569c-994b-4767-9426-9c1ba886cabe" containerName="main"
Sep 08 19:02:45.371433 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.371428    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6a569c-994b-4767-9426-9c1ba886cabe" containerName="main"
Sep 08 19:02:45.371574 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.371518    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6a569c-994b-4767-9426-9c1ba886cabe" containerName="main"
Sep 08 19:02:45.374878 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.374863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.376856 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.376837    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a4001b\"/\"rate-test-model-a4001b-kserve-self-signed-certs\""
Sep 08 19:02:45.383868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.383845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"]
Sep 08 19:02:45.397371 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.397328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:45.533966 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.533923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-dshm\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.534152 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.534021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-model-cache\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.534152 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.534089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhcpl\" (UniqueName: \"kubernetes.io/projected/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-kube-api-access-vhcpl\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.534152 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.534128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-home\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.534325 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.534195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tls-certs\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.534325 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.534247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tmp-dir\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635168 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-model-cache\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635168 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhcpl\" (UniqueName: \"kubernetes.io/projected/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-kube-api-access-vhcpl\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635168 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-home\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635476 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tls-certs\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635476 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tmp-dir\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635476 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-dshm\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635671 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tmp-dir\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635739 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-model-cache\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.635781 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.635736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-home\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.637523 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.637501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-dshm\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.637680 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.637662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tls-certs\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.642506 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.642489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhcpl\" (UniqueName: \"kubernetes.io/projected/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-kube-api-access-vhcpl\") pod \"rate-test-model-a4001b-kserve-7898bf9979-n9pcn\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") " pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.685664 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.685635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:45.776431 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.776404    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e6a569c-994b-4767-9426-9c1ba886cabe" containerID="f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6" exitCode=0
Sep 08 19:02:45.776824 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.776460    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8" event={"ID":"0e6a569c-994b-4767-9426-9c1ba886cabe","Type":"ContainerDied","Data":"f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6"}
Sep 08 19:02:45.776824 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.776489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8" event={"ID":"0e6a569c-994b-4767-9426-9c1ba886cabe","Type":"ContainerDied","Data":"8e9b3654f9129253073bc4dbecb692f1fc729ce7a1a1a34dabc108fc14ef6ed7"}
Sep 08 19:02:45.776824 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.776505    2579 scope.go:117] "RemoveContainer" containerID="f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6"
Sep 08 19:02:45.776824 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.776461    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"
Sep 08 19:02:45.786386 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.786366    2579 scope.go:117] "RemoveContainer" containerID="f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6"
Sep 08 19:02:45.786668 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:02:45.786647    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6\": container with ID starting with f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6 not found: ID does not exist" containerID="f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6"
Sep 08 19:02:45.786715 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.786676    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6"} err="failed to get container status \"f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6\": rpc error: code = NotFound desc = could not find container \"f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6\": container with ID starting with f959e9e000f5c946b509b44777d4198758825425f91fd76440ff13b4698403b6 not found: ID does not exist"
Sep 08 19:02:45.801370 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.801307    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx" podUID="fcfcf20d-ce79-4a37-8d75-355de6f26f80" containerName="main" containerID="cri-o://855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900" gracePeriod=45
Sep 08 19:02:45.802357 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.802324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"]
Sep 08 19:02:45.807829 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.807808    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/sub-test-model-a4001b-kserve-64dd5dfc6c-66pc8"]
Sep 08 19:02:45.956617 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:45.956589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"]
Sep 08 19:02:45.958308 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:02:45.958283    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5223a6a4_f765_4eb4_bd70_e8f2b9c5d709.slice/crio-a1e9ad00d981f35c81e0069661c36d3fa64c88aa53b89d76e88c4801cb98f3d0 WatchSource:0}: Error finding container a1e9ad00d981f35c81e0069661c36d3fa64c88aa53b89d76e88c4801cb98f3d0: Status 404 returned error can't find the container with id a1e9ad00d981f35c81e0069661c36d3fa64c88aa53b89d76e88c4801cb98f3d0
Sep 08 19:02:46.050950 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.050927    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:02:46.140215 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-dshm\") pod \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") "
Sep 08 19:02:46.140388 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140231    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-home\") pod \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") "
Sep 08 19:02:46.140388 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140291    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tls-certs\") pod \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") "
Sep 08 19:02:46.140388 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdslp\" (UniqueName: \"kubernetes.io/projected/fcfcf20d-ce79-4a37-8d75-355de6f26f80-kube-api-access-rdslp\") pod \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") "
Sep 08 19:02:46.140388 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tmp-dir\") pod \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") "
Sep 08 19:02:46.140592 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-model-cache\") pod \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\" (UID: \"fcfcf20d-ce79-4a37-8d75-355de6f26f80\") "
Sep 08 19:02:46.140592 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-home" (OuterVolumeSpecName: "home") pod "fcfcf20d-ce79-4a37-8d75-355de6f26f80" (UID: "fcfcf20d-ce79-4a37-8d75-355de6f26f80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:02:46.140722 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcfcf20d-ce79-4a37-8d75-355de6f26f80" (UID: "fcfcf20d-ce79-4a37-8d75-355de6f26f80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:02:46.140794 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140770    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcfcf20d-ce79-4a37-8d75-355de6f26f80" (UID: "fcfcf20d-ce79-4a37-8d75-355de6f26f80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:02:46.140868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140850    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:46.140915 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.140895    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:46.142399 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.142370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-dshm" (OuterVolumeSpecName: "dshm") pod "fcfcf20d-ce79-4a37-8d75-355de6f26f80" (UID: "fcfcf20d-ce79-4a37-8d75-355de6f26f80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:02:46.142504 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.142416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfcf20d-ce79-4a37-8d75-355de6f26f80-kube-api-access-rdslp" (OuterVolumeSpecName: "kube-api-access-rdslp") pod "fcfcf20d-ce79-4a37-8d75-355de6f26f80" (UID: "fcfcf20d-ce79-4a37-8d75-355de6f26f80"). InnerVolumeSpecName "kube-api-access-rdslp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:02:46.142504 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.142437    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcfcf20d-ce79-4a37-8d75-355de6f26f80" (UID: "fcfcf20d-ce79-4a37-8d75-355de6f26f80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:02:46.163394 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.163364    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6a569c-994b-4767-9426-9c1ba886cabe" path="/var/lib/kubelet/pods/0e6a569c-994b-4767-9426-9c1ba886cabe/volumes"
Sep 08 19:02:46.242177 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.242137    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdslp\" (UniqueName: \"kubernetes.io/projected/fcfcf20d-ce79-4a37-8d75-355de6f26f80-kube-api-access-rdslp\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:46.242177 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.242169    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:46.242177 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.242178    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfcf20d-ce79-4a37-8d75-355de6f26f80-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:46.242177 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.242186    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfcf20d-ce79-4a37-8d75-355de6f26f80-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:02:46.782963 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.782931    2579 generic.go:356] "Generic (PLEG): container finished" podID="fcfcf20d-ce79-4a37-8d75-355de6f26f80" containerID="855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900" exitCode=0
Sep 08 19:02:46.783438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.782976    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"
Sep 08 19:02:46.783438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.783023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx" event={"ID":"fcfcf20d-ce79-4a37-8d75-355de6f26f80","Type":"ContainerDied","Data":"855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900"}
Sep 08 19:02:46.783438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.783066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx" event={"ID":"fcfcf20d-ce79-4a37-8d75-355de6f26f80","Type":"ContainerDied","Data":"ec4165d8cde91f9aca8e05988ced9f7e563f03c115fe44034437dd4fbe7e9d4b"}
Sep 08 19:02:46.783438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.783084    2579 scope.go:117] "RemoveContainer" containerID="855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900"
Sep 08 19:02:46.785203 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.784855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn" event={"ID":"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709","Type":"ContainerStarted","Data":"4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef"}
Sep 08 19:02:46.785203 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.784890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn" event={"ID":"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709","Type":"ContainerStarted","Data":"a1e9ad00d981f35c81e0069661c36d3fa64c88aa53b89d76e88c4801cb98f3d0"}
Sep 08 19:02:46.791960 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.791943    2579 scope.go:117] "RemoveContainer" containerID="855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900"
Sep 08 19:02:46.792217 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:02:46.792197    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900\": container with ID starting with 855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900 not found: ID does not exist" containerID="855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900"
Sep 08 19:02:46.792267 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.792225    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900"} err="failed to get container status \"855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900\": rpc error: code = NotFound desc = could not find container \"855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900\": container with ID starting with 855b1be1c580dd5dde9cc2fed5057651f5732c81cdee1f6b97e6e098ab128900 not found: ID does not exist"
Sep 08 19:02:46.801818 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.801794    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"]
Sep 08 19:02:46.809262 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.809239    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/sub-test-model-525196-kserve-57595bf8dc-m5zcx"]
Sep 08 19:02:46.829254 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:46.829215    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn" podStartSLOduration=1.829203573 podStartE2EDuration="1.829203573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:46.826732288 +0000 UTC m=+1727.180825620" watchObservedRunningTime="2026-09-08 19:02:46.829203573 +0000 UTC m=+1727.183296905"
Sep 08 19:02:48.162752 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:48.162711    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcfcf20d-ce79-4a37-8d75-355de6f26f80" path="/var/lib/kubelet/pods/fcfcf20d-ce79-4a37-8d75-355de6f26f80/volumes"
Sep 08 19:02:51.523017 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:51.522988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:52.022948 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:52.022910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:52.086370 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:52.086315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:02:53.981438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:53.981402    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:02:55.686198 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:55.686162    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:55.686198 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:55.686208    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:55.698936 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:55.698910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:55.842907 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:55.842878    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:02:58.128986 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:02:58.128952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:00.943053 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:00.943019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:04.067444 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:04.067407    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:09.870606 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.870566    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"]
Sep 08 19:03:09.871174 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.871034    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfcf20d-ce79-4a37-8d75-355de6f26f80" containerName="main"
Sep 08 19:03:09.871174 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.871047    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfcf20d-ce79-4a37-8d75-355de6f26f80" containerName="main"
Sep 08 19:03:09.871174 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.871132    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfcf20d-ce79-4a37-8d75-355de6f26f80" containerName="main"
Sep 08 19:03:09.874487 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.874465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:09.879319 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.879292    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"default-dockercfg-tbv89\""
Sep 08 19:03:09.880021 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.879999    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"rate-test-model-525196-kserve-self-signed-certs\""
Sep 08 19:03:09.880275 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.880259    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"openshift-service-ca.crt\""
Sep 08 19:03:09.880645 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.880630    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-525196\"/\"kube-root-ca.crt\""
Sep 08 19:03:09.887702 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.887678    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"]
Sep 08 19:03:09.905894 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.905868    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:09.953094 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.953065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bggd8\" (UniqueName: \"kubernetes.io/projected/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-kube-api-access-bggd8\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:09.953199 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.953105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-home\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:09.953239 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.953205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tmp-dir\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:09.953239 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.953231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-model-cache\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:09.953312 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.953258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tls-certs\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:09.953368 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:09.953311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-dshm\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054280 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tmp-dir\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054459 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-model-cache\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054459 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tls-certs\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054459 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-dshm\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054459 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bggd8\" (UniqueName: \"kubernetes.io/projected/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-kube-api-access-bggd8\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054459 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-home\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054744 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tmp-dir\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054744 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-model-cache\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.054852 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.054764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-home\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.056576 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.056555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-dshm\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.056834 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.056817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tls-certs\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.062251 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.062232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bggd8\" (UniqueName: \"kubernetes.io/projected/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-kube-api-access-bggd8\") pod \"rate-test-model-525196-kserve-59bb75bb99-8mwff\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") " pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.183757 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.183681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:10.453598 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.453569    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"]
Sep 08 19:03:10.455582 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:03:10.455548    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb3e9f91_0cf9_44dc_a0e4_069e274871f2.slice/crio-c7c34db0ae113731e4120f8476f5ae424f891f3e89456e6e859035c2f4621fd6 WatchSource:0}: Error finding container c7c34db0ae113731e4120f8476f5ae424f891f3e89456e6e859035c2f4621fd6: Status 404 returned error can't find the container with id c7c34db0ae113731e4120f8476f5ae424f891f3e89456e6e859035c2f4621fd6
Sep 08 19:03:10.907858 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.907820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff" event={"ID":"cb3e9f91-0cf9-44dc-a0e4-069e274871f2","Type":"ContainerStarted","Data":"77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af"}
Sep 08 19:03:10.907858 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.907864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff" event={"ID":"cb3e9f91-0cf9-44dc-a0e4-069e274871f2","Type":"ContainerStarted","Data":"c7c34db0ae113731e4120f8476f5ae424f891f3e89456e6e859035c2f4621fd6"}
Sep 08 19:03:10.923376 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:10.923311    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff" podStartSLOduration=1.923298342 podStartE2EDuration="1.923298342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:03:10.922455959 +0000 UTC m=+1751.276549294" watchObservedRunningTime="2026-09-08 19:03:10.923298342 +0000 UTC m=+1751.277391675"
Sep 08 19:03:11.427029 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:11.426972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:13.954256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:13.954224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:15.475881 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:15.475844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:03:19.503250 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.503197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:19.534290 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.534258    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"]
Sep 08 19:03:19.534522 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.534493    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs" podUID="bc11e275-98f8-4ec0-a398-ddc55d23c630" containerName="payload-pre-processing" containerID="cri-o://d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4" gracePeriod=30
Sep 08 19:03:19.599366 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.599322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:19.687943 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.687915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:19.756112 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.755980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:19.787300 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.787272    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:03:19.846897 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.846861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfpvm\" (UniqueName: \"kubernetes.io/projected/bc11e275-98f8-4ec0-a398-ddc55d23c630-kube-api-access-xfpvm\") pod \"bc11e275-98f8-4ec0-a398-ddc55d23c630\" (UID: \"bc11e275-98f8-4ec0-a398-ddc55d23c630\") "
Sep 08 19:03:19.847096 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.846927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11e275-98f8-4ec0-a398-ddc55d23c630-plugins-config-volume\") pod \"bc11e275-98f8-4ec0-a398-ddc55d23c630\" (UID: \"bc11e275-98f8-4ec0-a398-ddc55d23c630\") "
Sep 08 19:03:19.847312 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.847290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc11e275-98f8-4ec0-a398-ddc55d23c630-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc11e275-98f8-4ec0-a398-ddc55d23c630" (UID: "bc11e275-98f8-4ec0-a398-ddc55d23c630"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:03:19.849274 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.849241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc11e275-98f8-4ec0-a398-ddc55d23c630-kube-api-access-xfpvm" (OuterVolumeSpecName: "kube-api-access-xfpvm") pod "bc11e275-98f8-4ec0-a398-ddc55d23c630" (UID: "bc11e275-98f8-4ec0-a398-ddc55d23c630"). InnerVolumeSpecName "kube-api-access-xfpvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:03:19.947831 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.947794    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfpvm\" (UniqueName: \"kubernetes.io/projected/bc11e275-98f8-4ec0-a398-ddc55d23c630-kube-api-access-xfpvm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:03:19.947831 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.947825    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11e275-98f8-4ec0-a398-ddc55d23c630-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:03:19.953458 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.953430    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc11e275-98f8-4ec0-a398-ddc55d23c630" containerID="d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4" exitCode=0
Sep 08 19:03:19.953627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.953486    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"
Sep 08 19:03:19.953627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.953505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs" event={"ID":"bc11e275-98f8-4ec0-a398-ddc55d23c630","Type":"ContainerDied","Data":"d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4"}
Sep 08 19:03:19.953627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.953548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs" event={"ID":"bc11e275-98f8-4ec0-a398-ddc55d23c630","Type":"ContainerDied","Data":"7639ef141216d387fe5b94fe273d5384a0da067d9b57671cf741b59146ca0f09"}
Sep 08 19:03:19.953627 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.953574    2579 scope.go:117] "RemoveContainer" containerID="d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4"
Sep 08 19:03:19.962457 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.962433    2579 scope.go:117] "RemoveContainer" containerID="d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4"
Sep 08 19:03:19.962718 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:03:19.962701    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4\": container with ID starting with d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4 not found: ID does not exist" containerID="d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4"
Sep 08 19:03:19.962771 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.962725    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4"} err="failed to get container status \"d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4\": rpc error: code = NotFound desc = could not find container \"d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4\": container with ID starting with d00879c538e1538052d6f4908b486b3147d6d30f2a7aa5b5fc8f05bfd71451a4 not found: ID does not exist"
Sep 08 19:03:19.982845 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.982813    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"]
Sep 08 19:03:19.985484 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:19.985458    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-965746-5df68cfd-qphbs"]
Sep 08 19:03:20.073451 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:20.073367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:20.163480 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:20.163451    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc11e275-98f8-4ec0-a398-ddc55d23c630" path="/var/lib/kubelet/pods/bc11e275-98f8-4ec0-a398-ddc55d23c630/volumes"
Sep 08 19:03:20.184356 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:20.184317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:20.184471 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:20.184375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:20.197073 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:20.197052    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:20.970961 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:20.970932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:03:25.937935 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:25.937903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:25.997457 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:25.997424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:26.205676 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:26.205590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:26.647795 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:26.647760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:33.883460 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:33.883426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:03:41.969983 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:41.969944    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"]
Sep 08 19:03:41.970577 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:41.970232    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99" podUID="1e4b1a7f-ef57-42fe-9b90-7dda15b15184" containerName="maas-api" containerID="cri-o://476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4" gracePeriod=30
Sep 08 19:03:42.003393 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.003352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:42.102878 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.102843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:42.181562 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.181532    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:42.218804 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.218778    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:03:42.244616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.244550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:42.365226 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.365193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-metrics-certs\") pod \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") "
Sep 08 19:03:42.365432 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.365262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-maas-api-tls\") pod \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") "
Sep 08 19:03:42.365432 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.365319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-openshift-service-ca\") pod \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") "
Sep 08 19:03:42.365432 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.365336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmmck\" (UniqueName: \"kubernetes.io/projected/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-kube-api-access-kmmck\") pod \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\" (UID: \"1e4b1a7f-ef57-42fe-9b90-7dda15b15184\") "
Sep 08 19:03:42.365768 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.365739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e4b1a7f-ef57-42fe-9b90-7dda15b15184" (UID: "1e4b1a7f-ef57-42fe-9b90-7dda15b15184"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:03:42.367291 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.367265    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e4b1a7f-ef57-42fe-9b90-7dda15b15184" (UID: "1e4b1a7f-ef57-42fe-9b90-7dda15b15184"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:03:42.367400 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.367362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-kube-api-access-kmmck" (OuterVolumeSpecName: "kube-api-access-kmmck") pod "1e4b1a7f-ef57-42fe-9b90-7dda15b15184" (UID: "1e4b1a7f-ef57-42fe-9b90-7dda15b15184"). InnerVolumeSpecName "kube-api-access-kmmck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:03:42.367464 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.367428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e4b1a7f-ef57-42fe-9b90-7dda15b15184" (UID: "1e4b1a7f-ef57-42fe-9b90-7dda15b15184"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:03:42.466312 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.466274    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:03:42.466312 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.466304    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:03:42.466312 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.466315    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:03:42.466312 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.466324    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmmck\" (UniqueName: \"kubernetes.io/projected/1e4b1a7f-ef57-42fe-9b90-7dda15b15184-kube-api-access-kmmck\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:03:42.570484 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:42.570404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:43.071255 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.071217    2579 generic.go:356] "Generic (PLEG): container finished" podID="1e4b1a7f-ef57-42fe-9b90-7dda15b15184" containerID="476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4" exitCode=0
Sep 08 19:03:43.071255 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.071258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99" event={"ID":"1e4b1a7f-ef57-42fe-9b90-7dda15b15184","Type":"ContainerDied","Data":"476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4"}
Sep 08 19:03:43.071735 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.071297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99" event={"ID":"1e4b1a7f-ef57-42fe-9b90-7dda15b15184","Type":"ContainerDied","Data":"8fc7b0bb510e0f8b83cc1ae115ca9349c0773d560f62070215660dcbbdc16e80"}
Sep 08 19:03:43.071735 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.071317    2579 scope.go:117] "RemoveContainer" containerID="476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4"
Sep 08 19:03:43.071735 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.071261    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"
Sep 08 19:03:43.080514 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.080495    2579 scope.go:117] "RemoveContainer" containerID="476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4"
Sep 08 19:03:43.080785 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:03:43.080768    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4\": container with ID starting with 476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4 not found: ID does not exist" containerID="476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4"
Sep 08 19:03:43.080825 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.080797    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4"} err="failed to get container status \"476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4\": rpc error: code = NotFound desc = could not find container \"476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4\": container with ID starting with 476458c8dda002aee1e3a1f088c6f50ad53b9b686b6c9f0f01cd94569d625fd4 not found: ID does not exist"
Sep 08 19:03:43.094118 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.094096    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"]
Sep 08 19:03:43.101662 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:43.101633    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee8372-8c55d64fc-rjz99"]
Sep 08 19:03:44.162653 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:44.162620    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e4b1a7f-ef57-42fe-9b90-7dda15b15184" path="/var/lib/kubelet/pods/1e4b1a7f-ef57-42fe-9b90-7dda15b15184/volumes"
Sep 08 19:03:48.621310 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:48.621274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:48.721445 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:48.721412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:48.903500 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:48.903386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:49.417813 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:49.417743    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:53.570745 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:53.570706    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:03:55.607253 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:55.607221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:57.751795 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:57.751759    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"]
Sep 08 19:03:57.770418 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:57.770379    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:58.679868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:58.679791    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"]
Sep 08 19:03:58.700271 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:58.700243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:59.134698 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:59.134662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:59.389688 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:59.389612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:59.489716 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:59.489685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:03:59.586634 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:03:59.586605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:00.078911 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.078880    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"]
Sep 08 19:04:00.082080 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.082051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:00.082209 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.082194    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-093570-29814900-85r5f"]
Sep 08 19:04:00.114164 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.114132    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"]
Sep 08 19:04:00.114455 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.114401    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s" podUID="779392ca-78f1-4bf0-9518-5909cc02efb1" containerName="payload-pre-processing" containerID="cri-o://a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f" gracePeriod=30
Sep 08 19:04:00.168474 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.168431    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d0c3bb1-f810-472b-95d1-4313110d3aff" path="/var/lib/kubelet/pods/9d0c3bb1-f810-472b-95d1-4313110d3aff/volumes"
Sep 08 19:04:00.184728 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.184697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:00.220179 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.220156    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 19:04:00.227541 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.227514    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 19:04:00.279954 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.279882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:00.346173 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.346146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:00.380475 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.380443    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 19:04:00.439386 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.439333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779392ca-78f1-4bf0-9518-5909cc02efb1-plugins-config-volume\") pod \"779392ca-78f1-4bf0-9518-5909cc02efb1\" (UID: \"779392ca-78f1-4bf0-9518-5909cc02efb1\") "
Sep 08 19:04:00.439584 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.439452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6mkc\" (UniqueName: \"kubernetes.io/projected/779392ca-78f1-4bf0-9518-5909cc02efb1-kube-api-access-c6mkc\") pod \"779392ca-78f1-4bf0-9518-5909cc02efb1\" (UID: \"779392ca-78f1-4bf0-9518-5909cc02efb1\") "
Sep 08 19:04:00.439791 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.439759    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779392ca-78f1-4bf0-9518-5909cc02efb1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "779392ca-78f1-4bf0-9518-5909cc02efb1" (UID: "779392ca-78f1-4bf0-9518-5909cc02efb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:04:00.441539 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.441508    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779392ca-78f1-4bf0-9518-5909cc02efb1-kube-api-access-c6mkc" (OuterVolumeSpecName: "kube-api-access-c6mkc") pod "779392ca-78f1-4bf0-9518-5909cc02efb1" (UID: "779392ca-78f1-4bf0-9518-5909cc02efb1"). InnerVolumeSpecName "kube-api-access-c6mkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:04:00.540902 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.540868    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779392ca-78f1-4bf0-9518-5909cc02efb1-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:00.540902 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.540894    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6mkc\" (UniqueName: \"kubernetes.io/projected/779392ca-78f1-4bf0-9518-5909cc02efb1-kube-api-access-c6mkc\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:00.641624 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:00.641538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:01.164112 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.164077    2579 generic.go:356] "Generic (PLEG): container finished" podID="779392ca-78f1-4bf0-9518-5909cc02efb1" containerID="a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f" exitCode=0
Sep 08 19:04:01.164325 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.164119    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"
Sep 08 19:04:01.164325 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.164170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s" event={"ID":"779392ca-78f1-4bf0-9518-5909cc02efb1","Type":"ContainerDied","Data":"a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f"}
Sep 08 19:04:01.164325 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.164224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s" event={"ID":"779392ca-78f1-4bf0-9518-5909cc02efb1","Type":"ContainerDied","Data":"58255c23764b173c1e68db1cb3dcbd48ee11f5976450abd903c1af586b03004b"}
Sep 08 19:04:01.164325 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.164258    2579 scope.go:117] "RemoveContainer" containerID="a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f"
Sep 08 19:04:01.173315 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.173295    2579 scope.go:117] "RemoveContainer" containerID="a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f"
Sep 08 19:04:01.173656 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:04:01.173636    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f\": container with ID starting with a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f not found: ID does not exist" containerID="a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f"
Sep 08 19:04:01.173705 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.173664    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f"} err="failed to get container status \"a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f\": rpc error: code = NotFound desc = could not find container \"a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f\": container with ID starting with a4893e59cadf338b2e1a294b50887633b800186cfedfb2b69e85312a9133bb2f not found: ID does not exist"
Sep 08 19:04:01.185101 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.185073    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"]
Sep 08 19:04:01.188442 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:01.188418    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-093570-7ff886446b-c4w8s"]
Sep 08 19:04:02.162790 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:02.162751    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779392ca-78f1-4bf0-9518-5909cc02efb1" path="/var/lib/kubelet/pods/779392ca-78f1-4bf0-9518-5909cc02efb1/volumes"
Sep 08 19:04:02.581636 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:02.581590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:02.653229 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:02.653195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:03.809352 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:03.809307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:03.883204 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:03.883170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:04.096997 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:04.096967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:04.258390 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:04.258327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:04.357450 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:04.357314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:06.520107 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:06.520037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:06.583990 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:06.583951    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:06.775615 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:06.775536    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:07.335213 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:07.335174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:08.369268 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:08.369233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:04:12.771946 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:12.771902    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn" podUID="5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" containerName="main" containerID="cri-o://4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef" gracePeriod=45
Sep 08 19:04:13.028746 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.028684    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:04:13.159275 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tmp-dir\") pod \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") "
Sep 08 19:04:13.159487 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tls-certs\") pod \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") "
Sep 08 19:04:13.159487 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159356    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-home\") pod \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") "
Sep 08 19:04:13.159581 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-dshm\") pod \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") "
Sep 08 19:04:13.159581 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhcpl\" (UniqueName: \"kubernetes.io/projected/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-kube-api-access-vhcpl\") pod \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") "
Sep 08 19:04:13.159687 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159586    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" (UID: "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:04:13.159687 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-model-cache\") pod \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\" (UID: \"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709\") "
Sep 08 19:04:13.159792 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-home" (OuterVolumeSpecName: "home") pod "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" (UID: "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:04:13.159935 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159911    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-model-cache" (OuterVolumeSpecName: "model-cache") pod "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" (UID: "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:04:13.160001 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.159987    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:13.160056 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.160006    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:13.161630 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.161602    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-kube-api-access-vhcpl" (OuterVolumeSpecName: "kube-api-access-vhcpl") pod "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" (UID: "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709"). InnerVolumeSpecName "kube-api-access-vhcpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:04:13.161948 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.161925    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" (UID: "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:04:13.162022 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.161928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-dshm" (OuterVolumeSpecName: "dshm") pod "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" (UID: "5223a6a4-f765-4eb4-bd70-e8f2b9c5d709"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:04:13.223887 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.223847    2579 generic.go:356] "Generic (PLEG): container finished" podID="5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" containerID="4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef" exitCode=0
Sep 08 19:04:13.223887 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.223889    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"
Sep 08 19:04:13.224091 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.223925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn" event={"ID":"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709","Type":"ContainerDied","Data":"4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef"}
Sep 08 19:04:13.224091 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.223969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn" event={"ID":"5223a6a4-f765-4eb4-bd70-e8f2b9c5d709","Type":"ContainerDied","Data":"a1e9ad00d981f35c81e0069661c36d3fa64c88aa53b89d76e88c4801cb98f3d0"}
Sep 08 19:04:13.224091 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.223985    2579 scope.go:117] "RemoveContainer" containerID="4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef"
Sep 08 19:04:13.233096 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.233077    2579 scope.go:117] "RemoveContainer" containerID="4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef"
Sep 08 19:04:13.233375 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:04:13.233356    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef\": container with ID starting with 4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef not found: ID does not exist" containerID="4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef"
Sep 08 19:04:13.233444 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.233388    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef"} err="failed to get container status \"4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef\": rpc error: code = NotFound desc = could not find container \"4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef\": container with ID starting with 4d8f5364ab1bf96ad11bf011b8e76568dbf25d858e6393e108e6a94f64ed37ef not found: ID does not exist"
Sep 08 19:04:13.243533 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.243505    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"]
Sep 08 19:04:13.248999 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.248978    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a4001b/rate-test-model-a4001b-kserve-7898bf9979-n9pcn"]
Sep 08 19:04:13.260805 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.260785    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:13.260872 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.260807    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:13.260872 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.260816    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhcpl\" (UniqueName: \"kubernetes.io/projected/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-kube-api-access-vhcpl\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:13.260872 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.260825    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:13.699443 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.699402    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff" podUID="cb3e9f91-0cf9-44dc-a0e4-069e274871f2" containerName="main" containerID="cri-o://77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af" gracePeriod=45
Sep 08 19:04:13.946105 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:13.946083    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:04:14.069152 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-dshm\") pod \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") "
Sep 08 19:04:14.069152 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tmp-dir\") pod \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") "
Sep 08 19:04:14.069152 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069144    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-model-cache\") pod \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") "
Sep 08 19:04:14.069474 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-home\") pod \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") "
Sep 08 19:04:14.069474 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tls-certs\") pod \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") "
Sep 08 19:04:14.069474 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bggd8\" (UniqueName: \"kubernetes.io/projected/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-kube-api-access-bggd8\") pod \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\" (UID: \"cb3e9f91-0cf9-44dc-a0e4-069e274871f2\") "
Sep 08 19:04:14.069623 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb3e9f91-0cf9-44dc-a0e4-069e274871f2" (UID: "cb3e9f91-0cf9-44dc-a0e4-069e274871f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:04:14.069623 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-home" (OuterVolumeSpecName: "home") pod "cb3e9f91-0cf9-44dc-a0e4-069e274871f2" (UID: "cb3e9f91-0cf9-44dc-a0e4-069e274871f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:04:14.069623 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069461    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb3e9f91-0cf9-44dc-a0e4-069e274871f2" (UID: "cb3e9f91-0cf9-44dc-a0e4-069e274871f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:04:14.069754 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069644    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:14.069754 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069661    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:14.069754 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.069676    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:14.071314 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.071281    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-dshm" (OuterVolumeSpecName: "dshm") pod "cb3e9f91-0cf9-44dc-a0e4-069e274871f2" (UID: "cb3e9f91-0cf9-44dc-a0e4-069e274871f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:04:14.071440 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.071330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-kube-api-access-bggd8" (OuterVolumeSpecName: "kube-api-access-bggd8") pod "cb3e9f91-0cf9-44dc-a0e4-069e274871f2" (UID: "cb3e9f91-0cf9-44dc-a0e4-069e274871f2"). InnerVolumeSpecName "kube-api-access-bggd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:04:14.071440 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.071363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb3e9f91-0cf9-44dc-a0e4-069e274871f2" (UID: "cb3e9f91-0cf9-44dc-a0e4-069e274871f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:04:14.163649 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.163618    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" path="/var/lib/kubelet/pods/5223a6a4-f765-4eb4-bd70-e8f2b9c5d709/volumes"
Sep 08 19:04:14.170085 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.170064    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:14.170085 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.170086    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bggd8\" (UniqueName: \"kubernetes.io/projected/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-kube-api-access-bggd8\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:14.170215 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.170095    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3e9f91-0cf9-44dc-a0e4-069e274871f2-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:14.230203 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.230169    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb3e9f91-0cf9-44dc-a0e4-069e274871f2" containerID="77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af" exitCode=0
Sep 08 19:04:14.230366 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.230209    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"
Sep 08 19:04:14.230366 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.230246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff" event={"ID":"cb3e9f91-0cf9-44dc-a0e4-069e274871f2","Type":"ContainerDied","Data":"77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af"}
Sep 08 19:04:14.230366 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.230280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff" event={"ID":"cb3e9f91-0cf9-44dc-a0e4-069e274871f2","Type":"ContainerDied","Data":"c7c34db0ae113731e4120f8476f5ae424f891f3e89456e6e859035c2f4621fd6"}
Sep 08 19:04:14.230366 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.230300    2579 scope.go:117] "RemoveContainer" containerID="77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af"
Sep 08 19:04:14.238680 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.238662    2579 scope.go:117] "RemoveContainer" containerID="77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af"
Sep 08 19:04:14.238966 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:04:14.238947    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af\": container with ID starting with 77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af not found: ID does not exist" containerID="77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af"
Sep 08 19:04:14.239026 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.238975    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af"} err="failed to get container status \"77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af\": rpc error: code = NotFound desc = could not find container \"77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af\": container with ID starting with 77cb6e4390a9a7615f9a4f80a4ee0e04fe90558e878f4890d1c02dec7f3f75af not found: ID does not exist"
Sep 08 19:04:14.243971 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.243945    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"]
Sep 08 19:04:14.247879 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:14.247858    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-525196/rate-test-model-525196-kserve-59bb75bb99-8mwff"]
Sep 08 19:04:16.163035 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:16.162999    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3e9f91-0cf9-44dc-a0e4-069e274871f2" path="/var/lib/kubelet/pods/cb3e9f91-0cf9-44dc-a0e4-069e274871f2/volumes"
Sep 08 19:04:17.556008 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.555970    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"]
Sep 08 19:04:17.556522 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.556261    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6" podUID="c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" containerName="maas-api" containerID="cri-o://f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7" gracePeriod=30
Sep 08 19:04:17.587964 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.587919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:17.611670 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.611637    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"]
Sep 08 19:04:17.611868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.611842    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d" podUID="ea635718-2e6e-44b2-b256-bd909bbddc41" containerName="payload-pre-processing" containerID="cri-o://144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a" gracePeriod=30
Sep 08 19:04:17.688900 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.688867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:17.775474 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.775433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:17.841648 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.841616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:17.879193 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.879165    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 19:04:17.900668 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:17.900638    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 19:04:18.007121 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007083    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbckd\" (UniqueName: \"kubernetes.io/projected/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-kube-api-access-kbckd\") pod \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") "
Sep 08 19:04:18.007291 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007155    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw2wc\" (UniqueName: \"kubernetes.io/projected/ea635718-2e6e-44b2-b256-bd909bbddc41-kube-api-access-nw2wc\") pod \"ea635718-2e6e-44b2-b256-bd909bbddc41\" (UID: \"ea635718-2e6e-44b2-b256-bd909bbddc41\") "
Sep 08 19:04:18.007291 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007194    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-metrics-certs\") pod \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") "
Sep 08 19:04:18.007291 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-openshift-service-ca\") pod \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") "
Sep 08 19:04:18.007530 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea635718-2e6e-44b2-b256-bd909bbddc41-plugins-config-volume\") pod \"ea635718-2e6e-44b2-b256-bd909bbddc41\" (UID: \"ea635718-2e6e-44b2-b256-bd909bbddc41\") "
Sep 08 19:04:18.007530 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007488    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls\") pod \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\" (UID: \"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9\") "
Sep 08 19:04:18.007652 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007628    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" (UID: "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:04:18.007764 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea635718-2e6e-44b2-b256-bd909bbddc41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea635718-2e6e-44b2-b256-bd909bbddc41" (UID: "ea635718-2e6e-44b2-b256-bd909bbddc41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:04:18.007865 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007848    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:18.007929 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.007870    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea635718-2e6e-44b2-b256-bd909bbddc41-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:18.009517 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.009489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" (UID: "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:04:18.009618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.009554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" (UID: "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:04:18.009618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.009560    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea635718-2e6e-44b2-b256-bd909bbddc41-kube-api-access-nw2wc" (OuterVolumeSpecName: "kube-api-access-nw2wc") pod "ea635718-2e6e-44b2-b256-bd909bbddc41" (UID: "ea635718-2e6e-44b2-b256-bd909bbddc41"). InnerVolumeSpecName "kube-api-access-nw2wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:04:18.009618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.009575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-kube-api-access-kbckd" (OuterVolumeSpecName: "kube-api-access-kbckd") pod "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" (UID: "c5c56237-1ea3-4da6-ae4f-92c54f3b87f9"). InnerVolumeSpecName "kube-api-access-kbckd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:04:18.108567 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.108527    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:18.108567 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.108561    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbckd\" (UniqueName: \"kubernetes.io/projected/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-kube-api-access-kbckd\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:18.108567 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.108572    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw2wc\" (UniqueName: \"kubernetes.io/projected/ea635718-2e6e-44b2-b256-bd909bbddc41-kube-api-access-nw2wc\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:18.108827 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.108582    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:04:18.124662 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.124632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:18.251477 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.251444    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" containerID="f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7" exitCode=0
Sep 08 19:04:18.251649 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.251493    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"
Sep 08 19:04:18.251649 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.251535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6" event={"ID":"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9","Type":"ContainerDied","Data":"f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7"}
Sep 08 19:04:18.251649 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.251590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6" event={"ID":"c5c56237-1ea3-4da6-ae4f-92c54f3b87f9","Type":"ContainerDied","Data":"61d47763f19a51c931b7e2285b129e48d67b62b84ebd6653ddb9f07f02643df1"}
Sep 08 19:04:18.251649 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.251620    2579 scope.go:117] "RemoveContainer" containerID="f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7"
Sep 08 19:04:18.253052 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.253033    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea635718-2e6e-44b2-b256-bd909bbddc41" containerID="144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a" exitCode=0
Sep 08 19:04:18.253150 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.253061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d" event={"ID":"ea635718-2e6e-44b2-b256-bd909bbddc41","Type":"ContainerDied","Data":"144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a"}
Sep 08 19:04:18.253150 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.253097    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"
Sep 08 19:04:18.253150 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.253110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d" event={"ID":"ea635718-2e6e-44b2-b256-bd909bbddc41","Type":"ContainerDied","Data":"0d029c4bc392633f7379beccdc1611d49f3ac670954347512b32fa712d480695"}
Sep 08 19:04:18.262203 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.262185    2579 scope.go:117] "RemoveContainer" containerID="f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7"
Sep 08 19:04:18.262500 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:04:18.262483    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7\": container with ID starting with f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7 not found: ID does not exist" containerID="f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7"
Sep 08 19:04:18.262542 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.262506    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7"} err="failed to get container status \"f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7\": rpc error: code = NotFound desc = could not find container \"f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7\": container with ID starting with f08887a7d40f62a1d359b654277edc4ab7173acfa65d8a5e85547a2aa3c3c9a7 not found: ID does not exist"
Sep 08 19:04:18.262542 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.262522    2579 scope.go:117] "RemoveContainer" containerID="144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a"
Sep 08 19:04:18.270688 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.270660    2579 scope.go:117] "RemoveContainer" containerID="144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a"
Sep 08 19:04:18.270824 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.270807    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"]
Sep 08 19:04:18.270978 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:04:18.270961    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a\": container with ID starting with 144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a not found: ID does not exist" containerID="144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a"
Sep 08 19:04:18.271030 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.270985    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a"} err="failed to get container status \"144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a\": rpc error: code = NotFound desc = could not find container \"144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a\": container with ID starting with 144cf480cf4a4c3f736a6d37d9fbb5cb8e0f850739f5bac51c570847e4e4733a not found: ID does not exist"
Sep 08 19:04:18.272552 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.272533    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c6f2ed-fff6d697b-n97b6"]
Sep 08 19:04:18.281238 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.281218    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"]
Sep 08 19:04:18.285361 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:18.285323    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c6f2ed-6d4d4c7d7d-22c5d"]
Sep 08 19:04:20.162675 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:20.162643    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" path="/var/lib/kubelet/pods/c5c56237-1ea3-4da6-ae4f-92c54f3b87f9/volumes"
Sep 08 19:04:20.163079 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:20.163005    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea635718-2e6e-44b2-b256-bd909bbddc41" path="/var/lib/kubelet/pods/ea635718-2e6e-44b2-b256-bd909bbddc41/volumes"
Sep 08 19:04:21.825992 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:21.825956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:23.948432 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:23.948355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:24.011485 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:24.011428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:24.273291 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:24.273169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:24.632843 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:24.632810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:24.670953 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:24.670915    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:04:27.516806 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:27.516753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:28.148381 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:28.148348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:28.443762 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:28.443677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:28.647729 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:28.647692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:31.985585 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:31.985555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:32.141646 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.141604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:32.910903 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.910871    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"]
Sep 08 19:04:32.911472 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911449    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc11e275-98f8-4ec0-a398-ddc55d23c630" containerName="payload-pre-processing"
Sep 08 19:04:32.911472 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911474    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc11e275-98f8-4ec0-a398-ddc55d23c630" containerName="payload-pre-processing"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911493    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea635718-2e6e-44b2-b256-bd909bbddc41" containerName="payload-pre-processing"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911503    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea635718-2e6e-44b2-b256-bd909bbddc41" containerName="payload-pre-processing"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911517    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" containerName="maas-api"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911526    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" containerName="maas-api"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911542    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779392ca-78f1-4bf0-9518-5909cc02efb1" containerName="payload-pre-processing"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911551    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="779392ca-78f1-4bf0-9518-5909cc02efb1" containerName="payload-pre-processing"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911560    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" containerName="main"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911569    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" containerName="main"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911585    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3e9f91-0cf9-44dc-a0e4-069e274871f2" containerName="main"
Sep 08 19:04:32.911616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911593    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3e9f91-0cf9-44dc-a0e4-069e274871f2" containerName="main"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911628    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4b1a7f-ef57-42fe-9b90-7dda15b15184" containerName="maas-api"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911637    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4b1a7f-ef57-42fe-9b90-7dda15b15184" containerName="maas-api"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911728    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc11e275-98f8-4ec0-a398-ddc55d23c630" containerName="payload-pre-processing"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911740    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="779392ca-78f1-4bf0-9518-5909cc02efb1" containerName="payload-pre-processing"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911751    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3e9f91-0cf9-44dc-a0e4-069e274871f2" containerName="main"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911760    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea635718-2e6e-44b2-b256-bd909bbddc41" containerName="payload-pre-processing"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911769    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c56237-1ea3-4da6-ae4f-92c54f3b87f9" containerName="maas-api"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911781    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4b1a7f-ef57-42fe-9b90-7dda15b15184" containerName="maas-api"
Sep 08 19:04:32.911918 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.911791    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5223a6a4-f765-4eb4-bd70-e8f2b9c5d709" containerName="main"
Sep 08 19:04:32.915142 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.915123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:32.917268 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.917249    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1a27c5-serving-cert\""
Sep 08 19:04:32.924528 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.924504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"]
Sep 08 19:04:32.942209 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.942181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:32.942326 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.942230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxfq\" (UniqueName: \"kubernetes.io/projected/31e86b11-059a-4e61-8a4e-92f3d074ed48-kube-api-access-jnxfq\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:32.942326 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.942272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e86b11-059a-4e61-8a4e-92f3d074ed48-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:32.942326 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.942306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:32.948052 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:32.948030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:33.043309 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.043279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.043786 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.043361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.043786 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.043387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxfq\" (UniqueName: \"kubernetes.io/projected/31e86b11-059a-4e61-8a4e-92f3d074ed48-kube-api-access-jnxfq\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.043786 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.043412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e86b11-059a-4e61-8a4e-92f3d074ed48-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.043786 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:04:33.043509    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-serving-cert: secret "maas-api-e2e-ipp-b-1a27c5-serving-cert" not found
Sep 08 19:04:33.043786 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:04:33.043585    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls podName:31e86b11-059a-4e61-8a4e-92f3d074ed48 nodeName:}" failed. No retries permitted until 2026-09-08 19:04:33.543562874 +0000 UTC m=+1833.897656203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls") pod "maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5" (UID: "31e86b11-059a-4e61-8a4e-92f3d074ed48") : secret "maas-api-e2e-ipp-b-1a27c5-serving-cert" not found
Sep 08 19:04:33.043996 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.043985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e86b11-059a-4e61-8a4e-92f3d074ed48-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.045754 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.045736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.051506 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.051470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:33.053595 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.053574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxfq\" (UniqueName: \"kubernetes.io/projected/31e86b11-059a-4e61-8a4e-92f3d074ed48-kube-api-access-jnxfq\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.388710 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.388674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:33.490424 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.490387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:33.547508 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.547479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.549997 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.549971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.563213 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.563187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:04:33.563877 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.563857    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"]
Sep 08 19:04:33.567415 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.567400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:33.569404 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.569385    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a27c5\""
Sep 08 19:04:33.569525 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.569464    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a27c5-dockercfg-96b9t\""
Sep 08 19:04:33.580920 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.580898    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"]
Sep 08 19:04:33.648296 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.648221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p\" (UID: \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:33.648459 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.648382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjj8w\" (UniqueName: \"kubernetes.io/projected/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-kube-api-access-mjj8w\") pod \"payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p\" (UID: \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:33.749577 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.749544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjj8w\" (UniqueName: \"kubernetes.io/projected/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-kube-api-access-mjj8w\") pod \"payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p\" (UID: \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:33.749724 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.749599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p\" (UID: \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:33.750296 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.750276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p\" (UID: \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:33.757271 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.757239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjj8w\" (UniqueName: \"kubernetes.io/projected/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-kube-api-access-mjj8w\") pod \"payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p\" (UID: \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:33.825966 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.825930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:33.884721 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:33.884687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:34.096076 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:34.096049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"]
Sep 08 19:04:34.098803 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:04:34.098774    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e86b11_059a_4e61_8a4e_92f3d074ed48.slice/crio-6247a2d6fd63e832f90253e996ec971c42dcb6bafaf6eed56213e132630934d2 WatchSource:0}: Error finding container 6247a2d6fd63e832f90253e996ec971c42dcb6bafaf6eed56213e132630934d2: Status 404 returned error can't find the container with id 6247a2d6fd63e832f90253e996ec971c42dcb6bafaf6eed56213e132630934d2
Sep 08 19:04:34.166664 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:34.166636    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"]
Sep 08 19:04:34.169032 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:04:34.169006    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dd01fc_b1ee_412f_b9f1_4e9f1256337c.slice/crio-b0c1945a9c608e712ab1f5bedcc1669c3efc3d41f58bd409be54953c411809d1 WatchSource:0}: Error finding container b0c1945a9c608e712ab1f5bedcc1669c3efc3d41f58bd409be54953c411809d1: Status 404 returned error can't find the container with id b0c1945a9c608e712ab1f5bedcc1669c3efc3d41f58bd409be54953c411809d1
Sep 08 19:04:34.331489 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:34.331452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5" event={"ID":"31e86b11-059a-4e61-8a4e-92f3d074ed48","Type":"ContainerStarted","Data":"6247a2d6fd63e832f90253e996ec971c42dcb6bafaf6eed56213e132630934d2"}
Sep 08 19:04:34.333127 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:34.333099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p" event={"ID":"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c","Type":"ContainerStarted","Data":"651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b"}
Sep 08 19:04:34.333250 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:34.333141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p" event={"ID":"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c","Type":"ContainerStarted","Data":"b0c1945a9c608e712ab1f5bedcc1669c3efc3d41f58bd409be54953c411809d1"}
Sep 08 19:04:34.333250 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:34.333174    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:34.347636 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:34.347588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p" podStartSLOduration=1.347574383 podStartE2EDuration="1.347574383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:04:34.346729376 +0000 UTC m=+1834.700822710" watchObservedRunningTime="2026-09-08 19:04:34.347574383 +0000 UTC m=+1834.701667715"
Sep 08 19:04:35.339275 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:35.339235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5" event={"ID":"31e86b11-059a-4e61-8a4e-92f3d074ed48","Type":"ContainerStarted","Data":"f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db"}
Sep 08 19:04:35.357061 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:35.357005    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5" podStartSLOduration=2.7719035290000003 podStartE2EDuration="3.356989598s" totalImagesPullingTime="585.086069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:04:35.354636589 +0000 UTC m=+1835.708729922" watchObservedRunningTime="2026-09-08 19:04:35.356989598 +0000 UTC m=+1835.711082931"
Sep 08 19:04:36.344653 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:36.344623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:36.987489 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:36.987452    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:04:42.354438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:42.354398    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:04:45.341920 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:45.341890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:04:56.710207 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:04:56.710174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:01.828252 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:01.828218    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p_c1dd01fc-b1ee-412f-b9f1-4e9f1256337c/payload-processing/0.log"
Sep 08 19:05:10.881744 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:10.881710    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"]
Sep 08 19:05:10.886251 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:10.886229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:10.888720 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:10.888691    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e77525\"/\"openshift-service-ca.crt\""
Sep 08 19:05:10.888862 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:10.888705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e77525\"/\"kube-root-ca.crt\""
Sep 08 19:05:10.889216 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:10.889199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e77525\"/\"ipp-route-e77525-kserve-self-signed-certs\""
Sep 08 19:05:10.889329 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:10.889252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e77525\"/\"default-dockercfg-rl48r\""
Sep 08 19:05:10.895105 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:10.895081    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"]
Sep 08 19:05:10.912130 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:10.912109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:11.005819 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.005788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-dshm\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.005963 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.005823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed674be2-ff9f-48ed-b767-349bbf7b8552-tls-certs\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.005963 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.005932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-model-cache\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.006044 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.005965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnw45\" (UniqueName: \"kubernetes.io/projected/ed674be2-ff9f-48ed-b767-349bbf7b8552-kube-api-access-hnw45\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.006044 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.005999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-tmp-dir\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.006044 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.006025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-home\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107309 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-model-cache\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107309 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnw45\" (UniqueName: \"kubernetes.io/projected/ed674be2-ff9f-48ed-b767-349bbf7b8552-kube-api-access-hnw45\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107545 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-tmp-dir\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107545 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-home\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107545 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-dshm\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107545 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed674be2-ff9f-48ed-b767-349bbf7b8552-tls-certs\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107791 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-model-cache\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107791 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-home\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.107901 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.107858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-tmp-dir\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.109661 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.109641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-dshm\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.109972 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.109955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed674be2-ff9f-48ed-b767-349bbf7b8552-tls-certs\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.115740 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.115717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnw45\" (UniqueName: \"kubernetes.io/projected/ed674be2-ff9f-48ed-b767-349bbf7b8552-kube-api-access-hnw45\") pod \"ipp-route-e77525-kserve-7f6bb7b455-v2db2\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") " pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.197321 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.197233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:11.469632 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.469494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"]
Sep 08 19:05:11.471915 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:05:11.471881    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded674be2_ff9f_48ed_b767_349bbf7b8552.slice/crio-e70f4c190e1dbf5715b78cf907ab513cac8f2b9c5c14e7ca621e89fe384fd179 WatchSource:0}: Error finding container e70f4c190e1dbf5715b78cf907ab513cac8f2b9c5c14e7ca621e89fe384fd179: Status 404 returned error can't find the container with id e70f4c190e1dbf5715b78cf907ab513cac8f2b9c5c14e7ca621e89fe384fd179
Sep 08 19:05:11.532321 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.532292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2" event={"ID":"ed674be2-ff9f-48ed-b767-349bbf7b8552","Type":"ContainerStarted","Data":"e70f4c190e1dbf5715b78cf907ab513cac8f2b9c5c14e7ca621e89fe384fd179"}
Sep 08 19:05:11.552214 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:11.552165    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2" podStartSLOduration=1.55214918 podStartE2EDuration="1.55214918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:05:11.551396283 +0000 UTC m=+1871.905489618" watchObservedRunningTime="2026-09-08 19:05:11.55214918 +0000 UTC m=+1871.906242513"
Sep 08 19:05:12.538939 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:12.538907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2" event={"ID":"ed674be2-ff9f-48ed-b767-349bbf7b8552","Type":"ContainerStarted","Data":"8338c7921af2af7913cc72900a586f46774a01b72f4c51a7f87d098732fa6f45"}
Sep 08 19:05:14.571481 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:14.571449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:05:21.197532 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:21.197493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:21.198017 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:21.197559    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:21.210407 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:21.210383    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:21.592947 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:21.592919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:05:36.175804 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:36.175767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:05:42.960435 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:42.960407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p_c1dd01fc-b1ee-412f-b9f1-4e9f1256337c/payload-processing/0.log"
Sep 08 19:05:43.247460 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:43.247370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:43.571374 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:43.571283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:43.727016 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:43.726984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:46.169933 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:46.169897    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:46.255398 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:46.255368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:47.375244 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.375209    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"]
Sep 08 19:05:47.380511 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.380487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.382593 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.382569    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-3a21b6-serving-cert\""
Sep 08 19:05:47.387474 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.387448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"]
Sep 08 19:05:47.408094 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.408060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:47.436085 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.436056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc419e-9872-458f-bfb8-bb2775e531fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.436257 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.436105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.436257 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.436130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.436257 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.436211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hs9\" (UniqueName: \"kubernetes.io/projected/40dc419e-9872-458f-bfb8-bb2775e531fc-kube-api-access-64hs9\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.467190 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.467156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:47.537286 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.537248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.537481 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.537298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.537481 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.537363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hs9\" (UniqueName: \"kubernetes.io/projected/40dc419e-9872-458f-bfb8-bb2775e531fc-kube-api-access-64hs9\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.537481 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:05:47.537408    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-serving-cert: secret "maas-api-e2e-ipp-cleanup-3a21b6-serving-cert" not found
Sep 08 19:05:47.537481 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.537449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc419e-9872-458f-bfb8-bb2775e531fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.537729 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:05:47.537490    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls podName:40dc419e-9872-458f-bfb8-bb2775e531fc nodeName:}" failed. No retries permitted until 2026-09-08 19:05:48.03746914 +0000 UTC m=+1908.391562478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq" (UID: "40dc419e-9872-458f-bfb8-bb2775e531fc") : secret "maas-api-e2e-ipp-cleanup-3a21b6-serving-cert" not found
Sep 08 19:05:47.538094 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.538069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc419e-9872-458f-bfb8-bb2775e531fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.539732 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.539712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.547848 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.547821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hs9\" (UniqueName: \"kubernetes.io/projected/40dc419e-9872-458f-bfb8-bb2775e531fc-kube-api-access-64hs9\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:47.606556 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.606518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:47.863548 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.863514    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:47.960043 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:47.960004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:05:48.041847 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:48.041811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:48.044413 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:48.044386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:48.294578 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:48.294492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:48.566291 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:48.566210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"]
Sep 08 19:05:48.570013 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:05:48.569985    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dc419e_9872_458f_bfb8_bb2775e531fc.slice/crio-cbb534a427572d734a9eed7ed2b88ee1c275a462b4852ca4b04eef6e887bd66e WatchSource:0}: Error finding container cbb534a427572d734a9eed7ed2b88ee1c275a462b4852ca4b04eef6e887bd66e: Status 404 returned error can't find the container with id cbb534a427572d734a9eed7ed2b88ee1c275a462b4852ca4b04eef6e887bd66e
Sep 08 19:05:48.571298 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:48.571280    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 19:05:48.713105 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:48.713067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq" event={"ID":"40dc419e-9872-458f-bfb8-bb2775e531fc","Type":"ContainerStarted","Data":"cbb534a427572d734a9eed7ed2b88ee1c275a462b4852ca4b04eef6e887bd66e"}
Sep 08 19:05:49.718936 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:49.718902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq" event={"ID":"40dc419e-9872-458f-bfb8-bb2775e531fc","Type":"ContainerStarted","Data":"8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb"}
Sep 08 19:05:49.719377 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:49.718978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:05:49.736319 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:49.736267    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq" podStartSLOduration=2.380854104 podStartE2EDuration="2.736252339s" totalImagesPullingTime="355.398235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:05:49.734700403 +0000 UTC m=+1910.088793737" watchObservedRunningTime="2026-09-08 19:05:49.736252339 +0000 UTC m=+1910.090345671"
Sep 08 19:05:51.177230 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:51.177194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:05:55.729170 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:05:55.729143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:06:11.229118 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:11.229075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:17.887807 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:17.887769    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"]
Sep 08 19:06:17.888508 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:17.888022    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq" podUID="40dc419e-9872-458f-bfb8-bb2775e531fc" containerName="maas-api" containerID="cri-o://8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb" gracePeriod=30
Sep 08 19:06:17.924227 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:17.924196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:18.030549 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.030504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:18.120037 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.120005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:18.129951 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.129927    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:06:18.182295 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.182225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:18.238012 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.237980    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc419e-9872-458f-bfb8-bb2775e531fc-openshift-service-ca\") pod \"40dc419e-9872-458f-bfb8-bb2775e531fc\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") "
Sep 08 19:06:18.238186 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.238125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64hs9\" (UniqueName: \"kubernetes.io/projected/40dc419e-9872-458f-bfb8-bb2775e531fc-kube-api-access-64hs9\") pod \"40dc419e-9872-458f-bfb8-bb2775e531fc\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") "
Sep 08 19:06:18.238256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.238189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls\") pod \"40dc419e-9872-458f-bfb8-bb2775e531fc\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") "
Sep 08 19:06:18.238256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.238222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-metrics-certs\") pod \"40dc419e-9872-458f-bfb8-bb2775e531fc\" (UID: \"40dc419e-9872-458f-bfb8-bb2775e531fc\") "
Sep 08 19:06:18.238473 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.238443    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dc419e-9872-458f-bfb8-bb2775e531fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40dc419e-9872-458f-bfb8-bb2775e531fc" (UID: "40dc419e-9872-458f-bfb8-bb2775e531fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:06:18.238629 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.238609    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc419e-9872-458f-bfb8-bb2775e531fc-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:06:18.240491 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.240453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dc419e-9872-458f-bfb8-bb2775e531fc-kube-api-access-64hs9" (OuterVolumeSpecName: "kube-api-access-64hs9") pod "40dc419e-9872-458f-bfb8-bb2775e531fc" (UID: "40dc419e-9872-458f-bfb8-bb2775e531fc"). InnerVolumeSpecName "kube-api-access-64hs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:06:18.240906 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.240885    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40dc419e-9872-458f-bfb8-bb2775e531fc" (UID: "40dc419e-9872-458f-bfb8-bb2775e531fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:06:18.241045 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.241030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "40dc419e-9872-458f-bfb8-bb2775e531fc" (UID: "40dc419e-9872-458f-bfb8-bb2775e531fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:06:18.339184 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.339146    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:06:18.339184 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.339172    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc419e-9872-458f-bfb8-bb2775e531fc-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:06:18.339184 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.339181    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hs9\" (UniqueName: \"kubernetes.io/projected/40dc419e-9872-458f-bfb8-bb2775e531fc-kube-api-access-64hs9\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:06:18.481434 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.481321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:18.872674 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.872637    2579 generic.go:356] "Generic (PLEG): container finished" podID="40dc419e-9872-458f-bfb8-bb2775e531fc" containerID="8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb" exitCode=0
Sep 08 19:06:18.872859 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.872682    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"
Sep 08 19:06:18.872859 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.872716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq" event={"ID":"40dc419e-9872-458f-bfb8-bb2775e531fc","Type":"ContainerDied","Data":"8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb"}
Sep 08 19:06:18.872859 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.872766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq" event={"ID":"40dc419e-9872-458f-bfb8-bb2775e531fc","Type":"ContainerDied","Data":"cbb534a427572d734a9eed7ed2b88ee1c275a462b4852ca4b04eef6e887bd66e"}
Sep 08 19:06:18.872859 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.872794    2579 scope.go:117] "RemoveContainer" containerID="8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb"
Sep 08 19:06:18.881990 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.881974    2579 scope.go:117] "RemoveContainer" containerID="8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb"
Sep 08 19:06:18.882246 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:06:18.882229    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb\": container with ID starting with 8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb not found: ID does not exist" containerID="8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb"
Sep 08 19:06:18.882298 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.882256    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb"} err="failed to get container status \"8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb\": rpc error: code = NotFound desc = could not find container \"8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb\": container with ID starting with 8c9a246f8be616e0a0c163ffcc5e100a992f059f1ab63f62aa58cfd1e31dfbdb not found: ID does not exist"
Sep 08 19:06:18.892170 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.892151    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"]
Sep 08 19:06:18.898559 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:18.898528    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3a21b6-5d67985d9c-bnppq"]
Sep 08 19:06:20.163973 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:20.163928    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40dc419e-9872-458f-bfb8-bb2775e531fc" path="/var/lib/kubelet/pods/40dc419e-9872-458f-bfb8-bb2775e531fc/volumes"
Sep 08 19:06:22.079261 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:22.079221    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:06:24.477594 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:24.477555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:24.537477 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:24.537447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:24.801737 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:24.801632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:28.955035 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:28.954995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:42.072801 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:42.072769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:06:44.349218 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:44.349150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:44.415288 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:44.415260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:44.480125 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:44.480091    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:44.544795 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:44.544762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:44.958583 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:44.958545    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:49.938799 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:49.938764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"]
Sep 08 19:06:49.985632 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:49.985602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:06:55.273748 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:06:55.273712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:07:00.400836 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:00.400806    2579 scope.go:117] "RemoveContainer" containerID="369cebc8771e1f5c230801808cce82751b89973b8cdadd359b2d867c546d1ac6"
Sep 08 19:07:04.958516 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:04.958477    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2" podUID="ed674be2-ff9f-48ed-b767-349bbf7b8552" containerName="main" containerID="cri-o://8338c7921af2af7913cc72900a586f46774a01b72f4c51a7f87d098732fa6f45" gracePeriod=45
Sep 08 19:07:05.106477 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.106445    2579 generic.go:356] "Generic (PLEG): container finished" podID="ed674be2-ff9f-48ed-b767-349bbf7b8552" containerID="8338c7921af2af7913cc72900a586f46774a01b72f4c51a7f87d098732fa6f45" exitCode=0
Sep 08 19:07:05.106642 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.106517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2" event={"ID":"ed674be2-ff9f-48ed-b767-349bbf7b8552","Type":"ContainerDied","Data":"8338c7921af2af7913cc72900a586f46774a01b72f4c51a7f87d098732fa6f45"}
Sep 08 19:07:05.202122 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.202098    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:07:05.274160 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.274076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed674be2-ff9f-48ed-b767-349bbf7b8552-tls-certs\") pod \"ed674be2-ff9f-48ed-b767-349bbf7b8552\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") "
Sep 08 19:07:05.274160 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.274125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-dshm\") pod \"ed674be2-ff9f-48ed-b767-349bbf7b8552\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") "
Sep 08 19:07:05.274160 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.274156    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-model-cache\") pod \"ed674be2-ff9f-48ed-b767-349bbf7b8552\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") "
Sep 08 19:07:05.274478 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.274179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnw45\" (UniqueName: \"kubernetes.io/projected/ed674be2-ff9f-48ed-b767-349bbf7b8552-kube-api-access-hnw45\") pod \"ed674be2-ff9f-48ed-b767-349bbf7b8552\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") "
Sep 08 19:07:05.274478 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.274256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-tmp-dir\") pod \"ed674be2-ff9f-48ed-b767-349bbf7b8552\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") "
Sep 08 19:07:05.274598 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.274486    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed674be2-ff9f-48ed-b767-349bbf7b8552" (UID: "ed674be2-ff9f-48ed-b767-349bbf7b8552"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:07:05.274658 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.274626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed674be2-ff9f-48ed-b767-349bbf7b8552" (UID: "ed674be2-ff9f-48ed-b767-349bbf7b8552"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:07:05.274819 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.274758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-home\") pod \"ed674be2-ff9f-48ed-b767-349bbf7b8552\" (UID: \"ed674be2-ff9f-48ed-b767-349bbf7b8552\") "
Sep 08 19:07:05.275112 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.275060    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:05.275240 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.275117    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:05.275240 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.275136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-home" (OuterVolumeSpecName: "home") pod "ed674be2-ff9f-48ed-b767-349bbf7b8552" (UID: "ed674be2-ff9f-48ed-b767-349bbf7b8552"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:07:05.276507 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.276485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed674be2-ff9f-48ed-b767-349bbf7b8552-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed674be2-ff9f-48ed-b767-349bbf7b8552" (UID: "ed674be2-ff9f-48ed-b767-349bbf7b8552"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:07:05.276609 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.276555    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed674be2-ff9f-48ed-b767-349bbf7b8552-kube-api-access-hnw45" (OuterVolumeSpecName: "kube-api-access-hnw45") pod "ed674be2-ff9f-48ed-b767-349bbf7b8552" (UID: "ed674be2-ff9f-48ed-b767-349bbf7b8552"). InnerVolumeSpecName "kube-api-access-hnw45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:07:05.276660 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.276632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-dshm" (OuterVolumeSpecName: "dshm") pod "ed674be2-ff9f-48ed-b767-349bbf7b8552" (UID: "ed674be2-ff9f-48ed-b767-349bbf7b8552"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:07:05.376520 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.376486    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:05.376520 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.376515    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnw45\" (UniqueName: \"kubernetes.io/projected/ed674be2-ff9f-48ed-b767-349bbf7b8552-kube-api-access-hnw45\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:05.376520 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.376525    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed674be2-ff9f-48ed-b767-349bbf7b8552-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:05.376752 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:05.376533    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed674be2-ff9f-48ed-b767-349bbf7b8552-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:06.112964 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:06.112930    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"
Sep 08 19:07:06.112964 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:06.112948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2" event={"ID":"ed674be2-ff9f-48ed-b767-349bbf7b8552","Type":"ContainerDied","Data":"e70f4c190e1dbf5715b78cf907ab513cac8f2b9c5c14e7ca621e89fe384fd179"}
Sep 08 19:07:06.113601 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:06.112993    2579 scope.go:117] "RemoveContainer" containerID="8338c7921af2af7913cc72900a586f46774a01b72f4c51a7f87d098732fa6f45"
Sep 08 19:07:06.133304 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:06.133279    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"]
Sep 08 19:07:06.136541 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:06.136519    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e77525/ipp-route-e77525-kserve-7f6bb7b455-v2db2"]
Sep 08 19:07:06.163136 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:06.163112    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed674be2-ff9f-48ed-b767-349bbf7b8552" path="/var/lib/kubelet/pods/ed674be2-ff9f-48ed-b767-349bbf7b8552/volumes"
Sep 08 19:07:24.091071 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:24.091033    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:26.481275 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:26.481241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:27.007514 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:27.007481    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:32.634356 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:32.634310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:32.700159 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:32.700126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:32.907303 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:32.907218    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:33.404723 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:33.404692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:43.671088 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.671052    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"]
Sep 08 19:07:43.671628 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.671295    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5" podUID="31e86b11-059a-4e61-8a4e-92f3d074ed48" containerName="maas-api" containerID="cri-o://f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db" gracePeriod=30
Sep 08 19:07:43.700749 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.700707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:43.722671 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.722639    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"]
Sep 08 19:07:43.722889 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.722864    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p" podUID="c1dd01fc-b1ee-412f-b9f1-4e9f1256337c" containerName="payload-processing" containerID="cri-o://651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b" gracePeriod=30
Sep 08 19:07:43.800849 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.800821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:43.882308 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.882170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:43.934883 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.934857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:43.965230 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.965202    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:07:43.995617 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:43.995595    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:07:44.054102 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.054071    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e86b11-059a-4e61-8a4e-92f3d074ed48-openshift-service-ca\") pod \"31e86b11-059a-4e61-8a4e-92f3d074ed48\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") "
Sep 08 19:07:44.054102 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.054108    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjj8w\" (UniqueName: \"kubernetes.io/projected/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-kube-api-access-mjj8w\") pod \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\" (UID: \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\") "
Sep 08 19:07:44.054396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.054127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnxfq\" (UniqueName: \"kubernetes.io/projected/31e86b11-059a-4e61-8a4e-92f3d074ed48-kube-api-access-jnxfq\") pod \"31e86b11-059a-4e61-8a4e-92f3d074ed48\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") "
Sep 08 19:07:44.054396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.054187    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-plugins-config-volume\") pod \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\" (UID: \"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c\") "
Sep 08 19:07:44.054396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.054258    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls\") pod \"31e86b11-059a-4e61-8a4e-92f3d074ed48\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") "
Sep 08 19:07:44.054396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.054316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-metrics-certs\") pod \"31e86b11-059a-4e61-8a4e-92f3d074ed48\" (UID: \"31e86b11-059a-4e61-8a4e-92f3d074ed48\") "
Sep 08 19:07:44.054669 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.054633    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e86b11-059a-4e61-8a4e-92f3d074ed48-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31e86b11-059a-4e61-8a4e-92f3d074ed48" (UID: "31e86b11-059a-4e61-8a4e-92f3d074ed48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:07:44.054669 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.054642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1dd01fc-b1ee-412f-b9f1-4e9f1256337c" (UID: "c1dd01fc-b1ee-412f-b9f1-4e9f1256337c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:07:44.056473 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.056432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-kube-api-access-mjj8w" (OuterVolumeSpecName: "kube-api-access-mjj8w") pod "c1dd01fc-b1ee-412f-b9f1-4e9f1256337c" (UID: "c1dd01fc-b1ee-412f-b9f1-4e9f1256337c"). InnerVolumeSpecName "kube-api-access-mjj8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:07:44.056587 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.056474    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e86b11-059a-4e61-8a4e-92f3d074ed48-kube-api-access-jnxfq" (OuterVolumeSpecName: "kube-api-access-jnxfq") pod "31e86b11-059a-4e61-8a4e-92f3d074ed48" (UID: "31e86b11-059a-4e61-8a4e-92f3d074ed48"). InnerVolumeSpecName "kube-api-access-jnxfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:07:44.056703 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.056687    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "31e86b11-059a-4e61-8a4e-92f3d074ed48" (UID: "31e86b11-059a-4e61-8a4e-92f3d074ed48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:07:44.056764 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.056742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31e86b11-059a-4e61-8a4e-92f3d074ed48" (UID: "31e86b11-059a-4e61-8a4e-92f3d074ed48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:07:44.155512 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.155477    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:44.155512 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.155505    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e86b11-059a-4e61-8a4e-92f3d074ed48-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:44.155512 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.155514    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e86b11-059a-4e61-8a4e-92f3d074ed48-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:44.155512 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.155524    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjj8w\" (UniqueName: \"kubernetes.io/projected/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-kube-api-access-mjj8w\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:44.155771 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.155532    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnxfq\" (UniqueName: \"kubernetes.io/projected/31e86b11-059a-4e61-8a4e-92f3d074ed48-kube-api-access-jnxfq\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:44.155771 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.155540    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:07:44.310731 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.310645    2579 generic.go:356] "Generic (PLEG): container finished" podID="31e86b11-059a-4e61-8a4e-92f3d074ed48" containerID="f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db" exitCode=0
Sep 08 19:07:44.310731 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.310687    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"
Sep 08 19:07:44.310942 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.310730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5" event={"ID":"31e86b11-059a-4e61-8a4e-92f3d074ed48","Type":"ContainerDied","Data":"f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db"}
Sep 08 19:07:44.310942 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.310796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5" event={"ID":"31e86b11-059a-4e61-8a4e-92f3d074ed48","Type":"ContainerDied","Data":"6247a2d6fd63e832f90253e996ec971c42dcb6bafaf6eed56213e132630934d2"}
Sep 08 19:07:44.310942 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.310831    2579 scope.go:117] "RemoveContainer" containerID="f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db"
Sep 08 19:07:44.312216 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.312193    2579 generic.go:356] "Generic (PLEG): container finished" podID="c1dd01fc-b1ee-412f-b9f1-4e9f1256337c" containerID="651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b" exitCode=0
Sep 08 19:07:44.312322 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.312236    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"
Sep 08 19:07:44.312322 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.312242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p" event={"ID":"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c","Type":"ContainerDied","Data":"651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b"}
Sep 08 19:07:44.312322 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.312276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p" event={"ID":"c1dd01fc-b1ee-412f-b9f1-4e9f1256337c","Type":"ContainerDied","Data":"b0c1945a9c608e712ab1f5bedcc1669c3efc3d41f58bd409be54953c411809d1"}
Sep 08 19:07:44.320233 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.320216    2579 scope.go:117] "RemoveContainer" containerID="f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db"
Sep 08 19:07:44.320547 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:07:44.320526    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db\": container with ID starting with f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db not found: ID does not exist" containerID="f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db"
Sep 08 19:07:44.320611 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.320553    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db"} err="failed to get container status \"f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db\": rpc error: code = NotFound desc = could not find container \"f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db\": container with ID starting with f0094b2f069941b07154f5a15282e74e6dae5fd7469ea7d90d16d524a5f674db not found: ID does not exist"
Sep 08 19:07:44.320611 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.320570    2579 scope.go:117] "RemoveContainer" containerID="651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b"
Sep 08 19:07:44.327295 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.327276    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"]
Sep 08 19:07:44.330172 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.330154    2579 scope.go:117] "RemoveContainer" containerID="651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b"
Sep 08 19:07:44.330607 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:07:44.330587    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b\": container with ID starting with 651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b not found: ID does not exist" containerID="651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b"
Sep 08 19:07:44.330697 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.330611    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b"} err="failed to get container status \"651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b\": rpc error: code = NotFound desc = could not find container \"651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b\": container with ID starting with 651e51447b603b025b380840c4213e75a1d1f2f15bf2d316326a325c7472638b not found: ID does not exist"
Sep 08 19:07:44.333189 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.333167    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a27c5-795fb7db67-24sp5"]
Sep 08 19:07:44.344545 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.344514    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"]
Sep 08 19:07:44.346955 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.346934    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a27c5-6f8b8b6d49-cks4p"]
Sep 08 19:07:44.940502 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:44.940409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:46.162387 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:46.162334    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e86b11-059a-4e61-8a4e-92f3d074ed48" path="/var/lib/kubelet/pods/31e86b11-059a-4e61-8a4e-92f3d074ed48/volumes"
Sep 08 19:07:46.162764 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:46.162732    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1dd01fc-b1ee-412f-b9f1-4e9f1256337c" path="/var/lib/kubelet/pods/c1dd01fc-b1ee-412f-b9f1-4e9f1256337c/volumes"
Sep 08 19:07:46.977954 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:46.977919    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:07:50.009240 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:50.009200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:50.109879 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:50.109844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:50.307252 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:50.307175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:50.828852 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:50.828824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:52.164249 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:52.164217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:55.917437 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:55.917407    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:07:55.979222 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:07:55.979192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:00.754256 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:00.754167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:01.346660 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.346625    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"]
Sep 08 19:08:01.346895 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.346854    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq" podUID="c9af8476-17bb-4bf4-8160-561042ccb373" containerName="maas-api" containerID="cri-o://abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95" gracePeriod=30
Sep 08 19:08:01.380316 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.380280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:01.406681 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.406645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"]
Sep 08 19:08:01.406899 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.406858    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh" podUID="b65fa616-2411-4ea8-9fdc-6fd33b0e698a" containerName="payload-pre-processing" containerID="cri-o://70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63" gracePeriod=30
Sep 08 19:08:01.480273 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.480233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:01.571455 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.571426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:01.641979 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.641950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:01.656028 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.656002    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 19:08:01.688999 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.688969    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 19:08:01.823797 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.823758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-metrics-certs\") pod \"c9af8476-17bb-4bf4-8160-561042ccb373\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") "
Sep 08 19:08:01.824283 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.823834    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af8476-17bb-4bf4-8160-561042ccb373-openshift-service-ca\") pod \"c9af8476-17bb-4bf4-8160-561042ccb373\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") "
Sep 08 19:08:01.824283 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.823922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xbnh\" (UniqueName: \"kubernetes.io/projected/c9af8476-17bb-4bf4-8160-561042ccb373-kube-api-access-5xbnh\") pod \"c9af8476-17bb-4bf4-8160-561042ccb373\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") "
Sep 08 19:08:01.824283 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.823953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x88n8\" (UniqueName: \"kubernetes.io/projected/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-kube-api-access-x88n8\") pod \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\" (UID: \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\") "
Sep 08 19:08:01.824283 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.824009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-plugins-config-volume\") pod \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\" (UID: \"b65fa616-2411-4ea8-9fdc-6fd33b0e698a\") "
Sep 08 19:08:01.824283 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.824040    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls\") pod \"c9af8476-17bb-4bf4-8160-561042ccb373\" (UID: \"c9af8476-17bb-4bf4-8160-561042ccb373\") "
Sep 08 19:08:01.824616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.824293    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9af8476-17bb-4bf4-8160-561042ccb373-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9af8476-17bb-4bf4-8160-561042ccb373" (UID: "c9af8476-17bb-4bf4-8160-561042ccb373"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:08:01.824616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.824544    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b65fa616-2411-4ea8-9fdc-6fd33b0e698a" (UID: "b65fa616-2411-4ea8-9fdc-6fd33b0e698a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:08:01.826148 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.826116    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-kube-api-access-x88n8" (OuterVolumeSpecName: "kube-api-access-x88n8") pod "b65fa616-2411-4ea8-9fdc-6fd33b0e698a" (UID: "b65fa616-2411-4ea8-9fdc-6fd33b0e698a"). InnerVolumeSpecName "kube-api-access-x88n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:08:01.826293 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.826271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9af8476-17bb-4bf4-8160-561042ccb373" (UID: "c9af8476-17bb-4bf4-8160-561042ccb373"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:08:01.826502 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.826483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9af8476-17bb-4bf4-8160-561042ccb373" (UID: "c9af8476-17bb-4bf4-8160-561042ccb373"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:08:01.826502 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.826492    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9af8476-17bb-4bf4-8160-561042ccb373-kube-api-access-5xbnh" (OuterVolumeSpecName: "kube-api-access-5xbnh") pod "c9af8476-17bb-4bf4-8160-561042ccb373" (UID: "c9af8476-17bb-4bf4-8160-561042ccb373"). InnerVolumeSpecName "kube-api-access-5xbnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:08:01.924985 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.924883    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xbnh\" (UniqueName: \"kubernetes.io/projected/c9af8476-17bb-4bf4-8160-561042ccb373-kube-api-access-5xbnh\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:01.924985 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.924914    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x88n8\" (UniqueName: \"kubernetes.io/projected/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-kube-api-access-x88n8\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:01.924985 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.924922    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65fa616-2411-4ea8-9fdc-6fd33b0e698a-plugins-config-volume\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:01.924985 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.924932    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:01.924985 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.924944    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9af8476-17bb-4bf4-8160-561042ccb373-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:01.924985 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.924953    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af8476-17bb-4bf4-8160-561042ccb373-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:01.945284 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:01.945255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:02.397531 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.397495    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9af8476-17bb-4bf4-8160-561042ccb373" containerID="abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95" exitCode=0
Sep 08 19:08:02.397728 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.397554    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"
Sep 08 19:08:02.397728 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.397582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq" event={"ID":"c9af8476-17bb-4bf4-8160-561042ccb373","Type":"ContainerDied","Data":"abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95"}
Sep 08 19:08:02.397728 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.397674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq" event={"ID":"c9af8476-17bb-4bf4-8160-561042ccb373","Type":"ContainerDied","Data":"0a2319609c3acb6ef200984a66b90090389b6c14ae7e7cea9acfa181c6931058"}
Sep 08 19:08:02.397728 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.397704    2579 scope.go:117] "RemoveContainer" containerID="abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95"
Sep 08 19:08:02.398999 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.398976    2579 generic.go:356] "Generic (PLEG): container finished" podID="b65fa616-2411-4ea8-9fdc-6fd33b0e698a" containerID="70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63" exitCode=0
Sep 08 19:08:02.399090 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.399021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh" event={"ID":"b65fa616-2411-4ea8-9fdc-6fd33b0e698a","Type":"ContainerDied","Data":"70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63"}
Sep 08 19:08:02.399090 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.399035    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"
Sep 08 19:08:02.399090 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.399058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh" event={"ID":"b65fa616-2411-4ea8-9fdc-6fd33b0e698a","Type":"ContainerDied","Data":"3c1e178b1d24022f06dd1eee731ee6eda94afc117834c934cb8b2a17284d6e55"}
Sep 08 19:08:02.407233 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.407218    2579 scope.go:117] "RemoveContainer" containerID="abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95"
Sep 08 19:08:02.407532 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:08:02.407514    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95\": container with ID starting with abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95 not found: ID does not exist" containerID="abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95"
Sep 08 19:08:02.407589 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.407539    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95"} err="failed to get container status \"abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95\": rpc error: code = NotFound desc = could not find container \"abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95\": container with ID starting with abdaf0501fb695ba406bd5bc373108b36c4c9a9bbaafcd3d69647480fb895e95 not found: ID does not exist"
Sep 08 19:08:02.407589 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.407553    2579 scope.go:117] "RemoveContainer" containerID="70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63"
Sep 08 19:08:02.417535 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.417508    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"]
Sep 08 19:08:02.419027 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.419001    2579 scope.go:117] "RemoveContainer" containerID="70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63"
Sep 08 19:08:02.419464 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:08:02.419435    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63\": container with ID starting with 70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63 not found: ID does not exist" containerID="70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63"
Sep 08 19:08:02.419581 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.419475    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63"} err="failed to get container status \"70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63\": rpc error: code = NotFound desc = could not find container \"70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63\": container with ID starting with 70052299f2dfbcf60629da57a6cd504268c641ef27321cbc5f1723d514bcdf63 not found: ID does not exist"
Sep 08 19:08:02.421262 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.421243    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4001b-845b6678f4-q95kh"]
Sep 08 19:08:02.431636 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.431580    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"]
Sep 08 19:08:02.433476 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:02.433456    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4001b-56d8bbc9f8-txxtq"]
Sep 08 19:08:04.163192 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:04.163158    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65fa616-2411-4ea8-9fdc-6fd33b0e698a" path="/var/lib/kubelet/pods/b65fa616-2411-4ea8-9fdc-6fd33b0e698a/volumes"
Sep 08 19:08:04.163586 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:04.163571    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9af8476-17bb-4bf4-8160-561042ccb373" path="/var/lib/kubelet/pods/c9af8476-17bb-4bf4-8160-561042ccb373/volumes"
Sep 08 19:08:04.568453 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:04.568368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:08:07.943063 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:07.943033    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:08.002703 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:08.002678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:08.437088 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:08.437060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:09.199904 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:09.199874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:19.517805 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.517764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"]
Sep 08 19:08:19.518352 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.517962    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v" podUID="08482b49-d555-47f9-86c3-41ffad57416e" containerName="maas-api" containerID="cri-o://d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b" gracePeriod=30
Sep 08 19:08:19.548237 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.548209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:19.620455 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.620425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:19.688084 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.688051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:19.748450 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.748418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:19.759227 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.759208    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 19:08:19.785297 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.785230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-metrics-certs\") pod \"08482b49-d555-47f9-86c3-41ffad57416e\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") "
Sep 08 19:08:19.785429 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.785303    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08482b49-d555-47f9-86c3-41ffad57416e-openshift-service-ca\") pod \"08482b49-d555-47f9-86c3-41ffad57416e\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") "
Sep 08 19:08:19.785429 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.785334    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-maas-api-tls\") pod \"08482b49-d555-47f9-86c3-41ffad57416e\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") "
Sep 08 19:08:19.785429 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.785403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8mlh\" (UniqueName: \"kubernetes.io/projected/08482b49-d555-47f9-86c3-41ffad57416e-kube-api-access-n8mlh\") pod \"08482b49-d555-47f9-86c3-41ffad57416e\" (UID: \"08482b49-d555-47f9-86c3-41ffad57416e\") "
Sep 08 19:08:19.785729 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.785702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08482b49-d555-47f9-86c3-41ffad57416e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08482b49-d555-47f9-86c3-41ffad57416e" (UID: "08482b49-d555-47f9-86c3-41ffad57416e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:08:19.790984 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.790507    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "08482b49-d555-47f9-86c3-41ffad57416e" (UID: "08482b49-d555-47f9-86c3-41ffad57416e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:08:19.790984 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.790901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08482b49-d555-47f9-86c3-41ffad57416e-kube-api-access-n8mlh" (OuterVolumeSpecName: "kube-api-access-n8mlh") pod "08482b49-d555-47f9-86c3-41ffad57416e" (UID: "08482b49-d555-47f9-86c3-41ffad57416e"). InnerVolumeSpecName "kube-api-access-n8mlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:08:19.795922 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.795782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08482b49-d555-47f9-86c3-41ffad57416e" (UID: "08482b49-d555-47f9-86c3-41ffad57416e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:08:19.886674 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.886642    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mlh\" (UniqueName: \"kubernetes.io/projected/08482b49-d555-47f9-86c3-41ffad57416e-kube-api-access-n8mlh\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:19.886674 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.886671    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:19.886857 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.886681    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08482b49-d555-47f9-86c3-41ffad57416e-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:19.886857 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:19.886690    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08482b49-d555-47f9-86c3-41ffad57416e-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:08:20.099931 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.099894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:20.489194 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.489099    2579 generic.go:356] "Generic (PLEG): container finished" podID="08482b49-d555-47f9-86c3-41ffad57416e" containerID="d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b" exitCode=0
Sep 08 19:08:20.489194 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.489161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v" event={"ID":"08482b49-d555-47f9-86c3-41ffad57416e","Type":"ContainerDied","Data":"d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b"}
Sep 08 19:08:20.489463 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.489205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v" event={"ID":"08482b49-d555-47f9-86c3-41ffad57416e","Type":"ContainerDied","Data":"a29dad42ada384c8e1b94e88f3b750068d945d896d31e3f90dc707d50ee729ff"}
Sep 08 19:08:20.489463 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.489231    2579 scope.go:117] "RemoveContainer" containerID="d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b"
Sep 08 19:08:20.489463 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.489165    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"
Sep 08 19:08:20.498300 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.498280    2579 scope.go:117] "RemoveContainer" containerID="d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b"
Sep 08 19:08:20.498615 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:08:20.498592    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b\": container with ID starting with d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b not found: ID does not exist" containerID="d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b"
Sep 08 19:08:20.498701 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.498624    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b"} err="failed to get container status \"d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b\": rpc error: code = NotFound desc = could not find container \"d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b\": container with ID starting with d414f3c80a387b67bc62d83f99a3853200799e980c597980ddea2a337598347b not found: ID does not exist"
Sep 08 19:08:20.505373 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.505336    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"]
Sep 08 19:08:20.509949 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:20.509926    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-525196-5544f8b985-cb68v"]
Sep 08 19:08:21.980322 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:21.980234    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:08:22.162258 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:22.162224    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08482b49-d555-47f9-86c3-41ffad57416e" path="/var/lib/kubelet/pods/08482b49-d555-47f9-86c3-41ffad57416e/volumes"
Sep 08 19:08:23.062113 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:23.062081    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:23.163125 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:23.163098    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:26.140677 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:26.140640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:26.194639 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:26.194611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:26.426910 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:26.426827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:31.269502 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:31.269467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:08:35.804357 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:35.804313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:36.956709 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:36.956656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:08:37.032046 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:08:37.032012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:09:00.255956 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:09:00.255927    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 19:09:00.265477 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:09:00.265458    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 19:09:51.739559 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:09:51.739523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:09:52.875112 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:09:52.875070    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:09:52.927545 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:09:52.927509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:09:54.477953 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:09:54.477914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:10:05.985844 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:10:05.985809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:10:18.477162 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:10:18.477122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:10:26.292143 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:10:26.292103    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:10:33.377977 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:10:33.377938    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:10:40.873020 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:10:40.872985    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:10:49.574285 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:10:49.574247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:10:52.895245 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:10:52.895209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:10:56.778393 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:10:56.778356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:11:04.668161 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:11:04.668123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:11:12.474379 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:11:12.474297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:12:20.931493 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:12:20.931461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:12:38.370354 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:12:38.370319    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:12:46.475421 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:12:46.475388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:12:54.779381 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:12:54.779335    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:13:00.970483 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970440    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"]
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970776    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dd01fc-b1ee-412f-b9f1-4e9f1256337c" containerName="payload-processing"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970787    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dd01fc-b1ee-412f-b9f1-4e9f1256337c" containerName="payload-processing"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970799    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e86b11-059a-4e61-8a4e-92f3d074ed48" containerName="maas-api"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970805    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e86b11-059a-4e61-8a4e-92f3d074ed48" containerName="maas-api"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970812    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed674be2-ff9f-48ed-b767-349bbf7b8552" containerName="main"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970817    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed674be2-ff9f-48ed-b767-349bbf7b8552" containerName="main"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970826    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40dc419e-9872-458f-bfb8-bb2775e531fc" containerName="maas-api"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970831    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40dc419e-9872-458f-bfb8-bb2775e531fc" containerName="maas-api"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970838    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65fa616-2411-4ea8-9fdc-6fd33b0e698a" containerName="payload-pre-processing"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970860    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65fa616-2411-4ea8-9fdc-6fd33b0e698a" containerName="payload-pre-processing"
Sep 08 19:13:00.970868 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970872    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9af8476-17bb-4bf4-8160-561042ccb373" containerName="maas-api"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970879    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9af8476-17bb-4bf4-8160-561042ccb373" containerName="maas-api"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970887    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08482b49-d555-47f9-86c3-41ffad57416e" containerName="maas-api"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970894    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08482b49-d555-47f9-86c3-41ffad57416e" containerName="maas-api"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970954    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08482b49-d555-47f9-86c3-41ffad57416e" containerName="maas-api"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970962    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1dd01fc-b1ee-412f-b9f1-4e9f1256337c" containerName="payload-processing"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970968    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40dc419e-9872-458f-bfb8-bb2775e531fc" containerName="maas-api"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970974    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed674be2-ff9f-48ed-b767-349bbf7b8552" containerName="main"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970981    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9af8476-17bb-4bf4-8160-561042ccb373" containerName="maas-api"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970987    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65fa616-2411-4ea8-9fdc-6fd33b0e698a" containerName="payload-pre-processing"
Sep 08 19:13:00.971384 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.970993    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e86b11-059a-4e61-8a4e-92f3d074ed48" containerName="maas-api"
Sep 08 19:13:00.974055 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.974037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:00.976061 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.976041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4960921b-kserve-self-signed-certs\""
Sep 08 19:13:00.980768 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:00.980746    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"]
Sep 08 19:13:01.002172 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.002151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:13:01.102472 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.102436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-dshm\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.102472 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.102478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tmp-dir\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.102689 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.102516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tls-certs\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.102689 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.102571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-home\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.102689 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.102623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-model-cache\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.102689 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.102646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlhnq\" (UniqueName: \"kubernetes.io/projected/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-kube-api-access-xlhnq\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.203983 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.203946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-dshm\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.203983 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.203987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tmp-dir\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.204176 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.204021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tls-certs\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.204222 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.204198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-home\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.204284 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.204269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-model-cache\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.204332 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.204305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlhnq\" (UniqueName: \"kubernetes.io/projected/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-kube-api-access-xlhnq\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.204529 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.204505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tmp-dir\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.204616 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.204562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-home\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.204680 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.204643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-model-cache\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.206333 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.206304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-dshm\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.206612 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.206595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tls-certs\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.214725 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.214702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlhnq\" (UniqueName: \"kubernetes.io/projected/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-kube-api-access-xlhnq\") pod \"e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") " pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.283570 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.283471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:01.552535 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.552508    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"]
Sep 08 19:13:01.555147 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:13:01.555119    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2d84ee_2230_4f3d_97fa_bbd26b3b3004.slice/crio-3a6d185a05cf4c7d7ac9bb94cd090e1f917431a9a28d7fe94056d4f0d4e5283a WatchSource:0}: Error finding container 3a6d185a05cf4c7d7ac9bb94cd090e1f917431a9a28d7fe94056d4f0d4e5283a: Status 404 returned error can't find the container with id 3a6d185a05cf4c7d7ac9bb94cd090e1f917431a9a28d7fe94056d4f0d4e5283a
Sep 08 19:13:01.761574 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.761537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2" event={"ID":"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004","Type":"ContainerStarted","Data":"72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f"}
Sep 08 19:13:01.761574 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.761580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2" event={"ID":"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004","Type":"ContainerStarted","Data":"3a6d185a05cf4c7d7ac9bb94cd090e1f917431a9a28d7fe94056d4f0d4e5283a"}
Sep 08 19:13:01.779218 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:01.779168    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2" podStartSLOduration=1.77915019 podStartE2EDuration="1.77915019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:13:01.777445677 +0000 UTC m=+2342.131539011" watchObservedRunningTime="2026-09-08 19:13:01.77915019 +0000 UTC m=+2342.133243523"
Sep 08 19:13:03.280017 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:03.279978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:13:11.075622 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.075586    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"]
Sep 08 19:13:11.079758 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.079731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.081913 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.081892    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4960921b-kserve-self-signed-certs\""
Sep 08 19:13:11.088229 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.088209    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"]
Sep 08 19:13:11.118625 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.118594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:13:11.207759 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.207723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-home\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.207889 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.207768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-model-cache\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.207889 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.207786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98rn\" (UniqueName: \"kubernetes.io/projected/cc4cdc1f-4035-4447-ba63-794f7375cf08-kube-api-access-p98rn\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.207989 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.207900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4cdc1f-4035-4447-ba63-794f7375cf08-tls-certs\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.207989 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.207932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-dshm\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.207989 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.207964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-tmp-dir\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.283663 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.283624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:11.283838 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.283679    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:11.296082 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.296054    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:11.309548 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.309511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-tmp-dir\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.309726 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.309584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-home\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.309726 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.309623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-model-cache\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.309726 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.309648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98rn\" (UniqueName: \"kubernetes.io/projected/cc4cdc1f-4035-4447-ba63-794f7375cf08-kube-api-access-p98rn\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.309726 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.309725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4cdc1f-4035-4447-ba63-794f7375cf08-tls-certs\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.309926 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.309790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-dshm\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.309982 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.309964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-tmp-dir\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.310020 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.309999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-model-cache\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.310071 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.310056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-home\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.312326 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.312302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-dshm\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.312648 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.312623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4cdc1f-4035-4447-ba63-794f7375cf08-tls-certs\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.318118 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.318097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98rn\" (UniqueName: \"kubernetes.io/projected/cc4cdc1f-4035-4447-ba63-794f7375cf08-kube-api-access-p98rn\") pod \"e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") " pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.391977 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.391947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:11.660908 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.660859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"]
Sep 08 19:13:11.663059 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:13:11.663033    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc4cdc1f_4035_4447_ba63_794f7375cf08.slice/crio-247ff19631843cc2b3066d2164255ab8fc52c04553afafa82d7314175566b529 WatchSource:0}: Error finding container 247ff19631843cc2b3066d2164255ab8fc52c04553afafa82d7314175566b529: Status 404 returned error can't find the container with id 247ff19631843cc2b3066d2164255ab8fc52c04553afafa82d7314175566b529
Sep 08 19:13:11.809659 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.809624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk" event={"ID":"cc4cdc1f-4035-4447-ba63-794f7375cf08","Type":"ContainerStarted","Data":"a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129"}
Sep 08 19:13:11.809815 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.809670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk" event={"ID":"cc4cdc1f-4035-4447-ba63-794f7375cf08","Type":"ContainerStarted","Data":"247ff19631843cc2b3066d2164255ab8fc52c04553afafa82d7314175566b529"}
Sep 08 19:13:11.820711 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.820687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:11.825777 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:11.825737    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk" podStartSLOduration=0.825722406 podStartE2EDuration="825.722406ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:13:11.824373842 +0000 UTC m=+2352.178467176" watchObservedRunningTime="2026-09-08 19:13:11.825722406 +0000 UTC m=+2352.179815740"
Sep 08 19:13:14.270055 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:14.270006    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:13:21.392242 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:21.392199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:21.392688 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:21.392264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:21.405019 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:21.404997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:21.866479 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:21.866451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:32.492267 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:32.492233    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"]
Sep 08 19:13:32.514906 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:32.514866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:13:34.270526 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:34.270496    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"]
Sep 08 19:13:34.287241 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:34.287212    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:13:34.671025 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:34.670989    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:13:43.469576 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:43.469533    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:13:47.512433 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.512391    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2" podUID="1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" containerName="main" containerID="cri-o://72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f" gracePeriod=45
Sep 08 19:13:47.764533 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.764470    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:47.859192 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tmp-dir\") pod \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") "
Sep 08 19:13:47.859398 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tls-certs\") pod \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") "
Sep 08 19:13:47.859398 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-home\") pod \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") "
Sep 08 19:13:47.859398 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" (UID: "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:13:47.859398 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlhnq\" (UniqueName: \"kubernetes.io/projected/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-kube-api-access-xlhnq\") pod \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") "
Sep 08 19:13:47.859398 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-dshm\") pod \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") "
Sep 08 19:13:47.859398 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-model-cache\") pod \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\" (UID: \"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004\") "
Sep 08 19:13:47.859728 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859531    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-home" (OuterVolumeSpecName: "home") pod "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" (UID: "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:13:47.859728 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859687    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" (UID: "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:13:47.859797 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859766    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:47.859797 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.859785    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:47.861464 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.861435    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-dshm" (OuterVolumeSpecName: "dshm") pod "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" (UID: "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:13:47.861573 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.861507    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-kube-api-access-xlhnq" (OuterVolumeSpecName: "kube-api-access-xlhnq") pod "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" (UID: "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004"). InnerVolumeSpecName "kube-api-access-xlhnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:13:47.861573 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.861539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" (UID: "1c2d84ee-2230-4f3d-97fa-bbd26b3b3004"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:13:47.960244 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.960203    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:47.960244 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.960235    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:47.960244 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.960250    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:47.960520 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.960261    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlhnq\" (UniqueName: \"kubernetes.io/projected/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004-kube-api-access-xlhnq\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:47.980656 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.980623    2579 generic.go:356] "Generic (PLEG): container finished" podID="1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" containerID="72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f" exitCode=0
Sep 08 19:13:47.980811 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.980661    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2" event={"ID":"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004","Type":"ContainerDied","Data":"72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f"}
Sep 08 19:13:47.980811 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.980678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"
Sep 08 19:13:47.980811 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.980688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2" event={"ID":"1c2d84ee-2230-4f3d-97fa-bbd26b3b3004","Type":"ContainerDied","Data":"3a6d185a05cf4c7d7ac9bb94cd090e1f917431a9a28d7fe94056d4f0d4e5283a"}
Sep 08 19:13:47.980811 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.980705    2579 scope.go:117] "RemoveContainer" containerID="72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f"
Sep 08 19:13:47.989438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.989420    2579 scope.go:117] "RemoveContainer" containerID="72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f"
Sep 08 19:13:47.989680 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:13:47.989662    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f\": container with ID starting with 72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f not found: ID does not exist" containerID="72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f"
Sep 08 19:13:47.989720 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:47.989688    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f"} err="failed to get container status \"72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f\": rpc error: code = NotFound desc = could not find container \"72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f\": container with ID starting with 72fe1c9b61edd4d5e968a0c8f50cd8cb1fdd6d0c55b6a12b0dd3e4b33bbb5d0f not found: ID does not exist"
Sep 08 19:13:48.000112 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:48.000088    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"]
Sep 08 19:13:48.004202 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:48.004180    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4960921b-kserve-855796fd9f-8hsf2"]
Sep 08 19:13:48.162843 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:48.162811    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" path="/var/lib/kubelet/pods/1c2d84ee-2230-4f3d-97fa-bbd26b3b3004/volumes"
Sep 08 19:13:49.290190 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.290155    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk" podUID="cc4cdc1f-4035-4447-ba63-794f7375cf08" containerName="main" containerID="cri-o://a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129" gracePeriod=45
Sep 08 19:13:49.536875 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.536847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:49.674763 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.674730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-dshm\") pod \"cc4cdc1f-4035-4447-ba63-794f7375cf08\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") "
Sep 08 19:13:49.674956 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.674775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p98rn\" (UniqueName: \"kubernetes.io/projected/cc4cdc1f-4035-4447-ba63-794f7375cf08-kube-api-access-p98rn\") pod \"cc4cdc1f-4035-4447-ba63-794f7375cf08\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") "
Sep 08 19:13:49.674956 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.674820    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4cdc1f-4035-4447-ba63-794f7375cf08-tls-certs\") pod \"cc4cdc1f-4035-4447-ba63-794f7375cf08\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") "
Sep 08 19:13:49.674956 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.674866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-model-cache\") pod \"cc4cdc1f-4035-4447-ba63-794f7375cf08\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") "
Sep 08 19:13:49.675144 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.674999    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-home\") pod \"cc4cdc1f-4035-4447-ba63-794f7375cf08\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") "
Sep 08 19:13:49.675144 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.675112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-tmp-dir\") pod \"cc4cdc1f-4035-4447-ba63-794f7375cf08\" (UID: \"cc4cdc1f-4035-4447-ba63-794f7375cf08\") "
Sep 08 19:13:49.675144 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.675115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc4cdc1f-4035-4447-ba63-794f7375cf08" (UID: "cc4cdc1f-4035-4447-ba63-794f7375cf08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:13:49.675312 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.675265    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-home" (OuterVolumeSpecName: "home") pod "cc4cdc1f-4035-4447-ba63-794f7375cf08" (UID: "cc4cdc1f-4035-4447-ba63-794f7375cf08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:13:49.675415 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.675386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc4cdc1f-4035-4447-ba63-794f7375cf08" (UID: "cc4cdc1f-4035-4447-ba63-794f7375cf08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:13:49.675493 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.675419    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-model-cache\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:49.675493 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.675437    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-home\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:49.677012 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.676989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4cdc1f-4035-4447-ba63-794f7375cf08-kube-api-access-p98rn" (OuterVolumeSpecName: "kube-api-access-p98rn") pod "cc4cdc1f-4035-4447-ba63-794f7375cf08" (UID: "cc4cdc1f-4035-4447-ba63-794f7375cf08"). InnerVolumeSpecName "kube-api-access-p98rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:13:49.677124 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.677040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc4cdc1f-4035-4447-ba63-794f7375cf08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc4cdc1f-4035-4447-ba63-794f7375cf08" (UID: "cc4cdc1f-4035-4447-ba63-794f7375cf08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:13:49.677124 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.677046    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-dshm" (OuterVolumeSpecName: "dshm") pod "cc4cdc1f-4035-4447-ba63-794f7375cf08" (UID: "cc4cdc1f-4035-4447-ba63-794f7375cf08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 19:13:49.776532 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.776494    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4cdc1f-4035-4447-ba63-794f7375cf08-tls-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:49.776532 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.776526    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-tmp-dir\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:49.776532 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.776534    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4cdc1f-4035-4447-ba63-794f7375cf08-dshm\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:49.776756 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.776542    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p98rn\" (UniqueName: \"kubernetes.io/projected/cc4cdc1f-4035-4447-ba63-794f7375cf08-kube-api-access-p98rn\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:13:49.992743 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.992655    2579 generic.go:356] "Generic (PLEG): container finished" podID="cc4cdc1f-4035-4447-ba63-794f7375cf08" containerID="a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129" exitCode=0
Sep 08 19:13:49.992743 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.992699    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"
Sep 08 19:13:49.992940 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.992735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk" event={"ID":"cc4cdc1f-4035-4447-ba63-794f7375cf08","Type":"ContainerDied","Data":"a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129"}
Sep 08 19:13:49.992940 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.992778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk" event={"ID":"cc4cdc1f-4035-4447-ba63-794f7375cf08","Type":"ContainerDied","Data":"247ff19631843cc2b3066d2164255ab8fc52c04553afafa82d7314175566b529"}
Sep 08 19:13:49.992940 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:49.992795    2579 scope.go:117] "RemoveContainer" containerID="a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129"
Sep 08 19:13:50.001486 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:50.001468    2579 scope.go:117] "RemoveContainer" containerID="a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129"
Sep 08 19:13:50.001745 ip-10-0-140-26 kubenswrapper[2579]: E0908 19:13:50.001727    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129\": container with ID starting with a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129 not found: ID does not exist" containerID="a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129"
Sep 08 19:13:50.001794 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:50.001753    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129"} err="failed to get container status \"a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129\": rpc error: code = NotFound desc = could not find container \"a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129\": container with ID starting with a76fa0f3112a2926c1456b17f92e5b022a79230616ceb17240d1d896dbb9a129 not found: ID does not exist"
Sep 08 19:13:50.014847 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:50.014821    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"]
Sep 08 19:13:50.018463 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:50.018445    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4960921b-kserve-77dbb569c5-tlfdk"]
Sep 08 19:13:50.162847 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:50.162812    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc4cdc1f-4035-4447-ba63-794f7375cf08" path="/var/lib/kubelet/pods/cc4cdc1f-4035-4447-ba63-794f7375cf08/volumes"
Sep 08 19:13:50.766862 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:50.766830    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:13:57.277354 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.277309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:13:57.579678 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.579655    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"]
Sep 08 19:13:57.580016 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.580004    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc4cdc1f-4035-4447-ba63-794f7375cf08" containerName="main"
Sep 08 19:13:57.580064 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.580018    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc4cdc1f-4035-4447-ba63-794f7375cf08" containerName="main"
Sep 08 19:13:57.580064 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.580033    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" containerName="main"
Sep 08 19:13:57.580064 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.580038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" containerName="main"
Sep 08 19:13:57.580162 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.580090    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc4cdc1f-4035-4447-ba63-794f7375cf08" containerName="main"
Sep 08 19:13:57.580162 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.580102    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2d84ee-2230-4f3d-97fa-bbd26b3b3004" containerName="main"
Sep 08 19:13:57.584655 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.584638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.587336 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.587314    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l9rcm\""
Sep 08 19:13:57.587485 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.587404    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 19:13:57.587485 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.587421    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 19:13:57.587485 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.587403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 19:13:57.587485 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.587403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 19:13:57.593386 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.593362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"]
Sep 08 19:13:57.747889 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.747853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-metrics-certs\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.747889 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.747891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zkn\" (UniqueName: \"kubernetes.io/projected/12ca210a-7b21-4499-9c17-f6b1f279551a-kube-api-access-54zkn\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.748112 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.748018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-maas-api-tls\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.748112 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.748054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ca210a-7b21-4499-9c17-f6b1f279551a-openshift-service-ca\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.849396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.849284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-maas-api-tls\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.849396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.849329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ca210a-7b21-4499-9c17-f6b1f279551a-openshift-service-ca\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.849396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.849396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-metrics-certs\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.849685 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.849412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zkn\" (UniqueName: \"kubernetes.io/projected/12ca210a-7b21-4499-9c17-f6b1f279551a-kube-api-access-54zkn\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.850122 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.850100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ca210a-7b21-4499-9c17-f6b1f279551a-openshift-service-ca\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.851788 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.851766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-metrics-certs\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.851788 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.851787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-maas-api-tls\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.856462 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.856432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zkn\" (UniqueName: \"kubernetes.io/projected/12ca210a-7b21-4499-9c17-f6b1f279551a-kube-api-access-54zkn\") pod \"maas-api-dd87ddf4c-v7nzt\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:57.896452 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:57.896420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:58.164725 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:58.164690    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"]
Sep 08 19:13:58.167472 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:13:58.167444    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ca210a_7b21_4499_9c17_f6b1f279551a.slice/crio-cd7b9473c8e19cfd56ef3836bd64aaf3a87af02cd691c771d86d6f58d86708e1 WatchSource:0}: Error finding container cd7b9473c8e19cfd56ef3836bd64aaf3a87af02cd691c771d86d6f58d86708e1: Status 404 returned error can't find the container with id cd7b9473c8e19cfd56ef3836bd64aaf3a87af02cd691c771d86d6f58d86708e1
Sep 08 19:13:58.168738 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:58.168722    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 19:13:59.036193 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:59.036102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt" event={"ID":"12ca210a-7b21-4499-9c17-f6b1f279551a","Type":"ContainerStarted","Data":"583298f789d8265dc61737226daf1c1013f1dee597bedbb00648c7e16c2385ef"}
Sep 08 19:13:59.036193 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:59.036160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt" event={"ID":"12ca210a-7b21-4499-9c17-f6b1f279551a","Type":"ContainerStarted","Data":"cd7b9473c8e19cfd56ef3836bd64aaf3a87af02cd691c771d86d6f58d86708e1"}
Sep 08 19:13:59.036736 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:59.036207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:13:59.052429 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:13:59.052381    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt" podStartSLOduration=1.557837534 podStartE2EDuration="2.052368758s" totalImagesPullingTime="494.531224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:13:59.050689544 +0000 UTC m=+2399.404782903" watchObservedRunningTime="2026-09-08 19:13:59.052368758 +0000 UTC m=+2399.406462087"
Sep 08 19:14:00.290538 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:00.290508    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 19:14:00.299108 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:00.299088    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m6qmp_4396b3d0-3459-453a-80f8-468b02bee7cb/ovn-acl-logging/0.log"
Sep 08 19:14:04.823729 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.823694    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"]
Sep 08 19:14:04.824354 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.823894    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt" podUID="12ca210a-7b21-4499-9c17-f6b1f279551a" containerName="maas-api" containerID="cri-o://583298f789d8265dc61737226daf1c1013f1dee597bedbb00648c7e16c2385ef" gracePeriod=30
Sep 08 19:14:04.830817 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.830790    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:14:04.865125 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.865097    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"]
Sep 08 19:14:04.868660 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.868639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:04.881350 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.881316    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"]
Sep 08 19:14:04.912629 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.912606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/435ff9d4-9617-41fb-ba25-6cffe14e9843-openshift-service-ca\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:04.912752 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.912648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/435ff9d4-9617-41fb-ba25-6cffe14e9843-metrics-certs\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:04.912752 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.912700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2q6\" (UniqueName: \"kubernetes.io/projected/435ff9d4-9617-41fb-ba25-6cffe14e9843-kube-api-access-np2q6\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:04.912826 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.912764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435ff9d4-9617-41fb-ba25-6cffe14e9843-maas-api-tls\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:04.978675 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:04.978641    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:14:05.014492 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.014456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2q6\" (UniqueName: \"kubernetes.io/projected/435ff9d4-9617-41fb-ba25-6cffe14e9843-kube-api-access-np2q6\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.014682 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.014557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435ff9d4-9617-41fb-ba25-6cffe14e9843-maas-api-tls\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.014682 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.014617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/435ff9d4-9617-41fb-ba25-6cffe14e9843-openshift-service-ca\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.014819 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.014799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/435ff9d4-9617-41fb-ba25-6cffe14e9843-metrics-certs\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.015441 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.015394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/435ff9d4-9617-41fb-ba25-6cffe14e9843-openshift-service-ca\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.017595 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.017567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/435ff9d4-9617-41fb-ba25-6cffe14e9843-metrics-certs\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.017815 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.017795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435ff9d4-9617-41fb-ba25-6cffe14e9843-maas-api-tls\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.022438 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.022416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2q6\" (UniqueName: \"kubernetes.io/projected/435ff9d4-9617-41fb-ba25-6cffe14e9843-kube-api-access-np2q6\") pod \"maas-api-79cd597bc5-p4lzw\" (UID: \"435ff9d4-9617-41fb-ba25-6cffe14e9843\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.070227 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.070200    2579 generic.go:356] "Generic (PLEG): container finished" podID="12ca210a-7b21-4499-9c17-f6b1f279551a" containerID="583298f789d8265dc61737226daf1c1013f1dee597bedbb00648c7e16c2385ef" exitCode=0
Sep 08 19:14:05.070369 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.070238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt" event={"ID":"12ca210a-7b21-4499-9c17-f6b1f279551a","Type":"ContainerDied","Data":"583298f789d8265dc61737226daf1c1013f1dee597bedbb00648c7e16c2385ef"}
Sep 08 19:14:05.070369 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.070290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt" event={"ID":"12ca210a-7b21-4499-9c17-f6b1f279551a","Type":"ContainerDied","Data":"cd7b9473c8e19cfd56ef3836bd64aaf3a87af02cd691c771d86d6f58d86708e1"}
Sep 08 19:14:05.070369 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.070309    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd7b9473c8e19cfd56ef3836bd64aaf3a87af02cd691c771d86d6f58d86708e1"
Sep 08 19:14:05.072746 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.072728    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:14:05.115395 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.115371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-maas-api-tls\") pod \"12ca210a-7b21-4499-9c17-f6b1f279551a\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") "
Sep 08 19:14:05.115528 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.115406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ca210a-7b21-4499-9c17-f6b1f279551a-openshift-service-ca\") pod \"12ca210a-7b21-4499-9c17-f6b1f279551a\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") "
Sep 08 19:14:05.115528 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.115462    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54zkn\" (UniqueName: \"kubernetes.io/projected/12ca210a-7b21-4499-9c17-f6b1f279551a-kube-api-access-54zkn\") pod \"12ca210a-7b21-4499-9c17-f6b1f279551a\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") "
Sep 08 19:14:05.115528 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.115481    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-metrics-certs\") pod \"12ca210a-7b21-4499-9c17-f6b1f279551a\" (UID: \"12ca210a-7b21-4499-9c17-f6b1f279551a\") "
Sep 08 19:14:05.115832 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.115795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ca210a-7b21-4499-9c17-f6b1f279551a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12ca210a-7b21-4499-9c17-f6b1f279551a" (UID: "12ca210a-7b21-4499-9c17-f6b1f279551a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 19:14:05.117658 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.117625    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ca210a-7b21-4499-9c17-f6b1f279551a-kube-api-access-54zkn" (OuterVolumeSpecName: "kube-api-access-54zkn") pod "12ca210a-7b21-4499-9c17-f6b1f279551a" (UID: "12ca210a-7b21-4499-9c17-f6b1f279551a"). InnerVolumeSpecName "kube-api-access-54zkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 19:14:05.117658 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.117643    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12ca210a-7b21-4499-9c17-f6b1f279551a" (UID: "12ca210a-7b21-4499-9c17-f6b1f279551a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:14:05.117803 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.117666    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12ca210a-7b21-4499-9c17-f6b1f279551a" (UID: "12ca210a-7b21-4499-9c17-f6b1f279551a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 19:14:05.208325 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.208289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:05.216925 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.216905    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-maas-api-tls\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:14:05.216991 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.216929    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ca210a-7b21-4499-9c17-f6b1f279551a-openshift-service-ca\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:14:05.216991 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.216953    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54zkn\" (UniqueName: \"kubernetes.io/projected/12ca210a-7b21-4499-9c17-f6b1f279551a-kube-api-access-54zkn\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:14:05.216991 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.216963    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ca210a-7b21-4499-9c17-f6b1f279551a-metrics-certs\") on node \"ip-10-0-140-26.ec2.internal\" DevicePath \"\""
Sep 08 19:14:05.478418 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:05.478396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"]
Sep 08 19:14:05.480397 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:14:05.480371    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435ff9d4_9617_41fb_ba25_6cffe14e9843.slice/crio-408516db05a38731d403c26552186f4195cd088d90ec50625f7903ee57319260 WatchSource:0}: Error finding container 408516db05a38731d403c26552186f4195cd088d90ec50625f7903ee57319260: Status 404 returned error can't find the container with id 408516db05a38731d403c26552186f4195cd088d90ec50625f7903ee57319260
Sep 08 19:14:06.076897 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:06.076807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw" event={"ID":"435ff9d4-9617-41fb-ba25-6cffe14e9843","Type":"ContainerStarted","Data":"cf0e49bbd1e24f8c815f5f1f0e9e6a2fe256dc8d1841b351613703a46ffb602d"}
Sep 08 19:14:06.076897 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:06.076872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw" event={"ID":"435ff9d4-9617-41fb-ba25-6cffe14e9843","Type":"ContainerStarted","Data":"408516db05a38731d403c26552186f4195cd088d90ec50625f7903ee57319260"}
Sep 08 19:14:06.076897 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:06.076884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"
Sep 08 19:14:06.077336 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:06.077110    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:06.099367 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:06.097814    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw" podStartSLOduration=1.7972075140000001 podStartE2EDuration="2.097797512s" totalImagesPullingTime="300.589998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:14:06.095709272 +0000 UTC m=+2406.449802602" watchObservedRunningTime="2026-09-08 19:14:06.097797512 +0000 UTC m=+2406.451890847"
Sep 08 19:14:06.123056 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:06.123026    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"]
Sep 08 19:14:06.126809 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:06.126782    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-v7nzt"]
Sep 08 19:14:06.162563 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:06.162528    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ca210a-7b21-4499-9c17-f6b1f279551a" path="/var/lib/kubelet/pods/12ca210a-7b21-4499-9c17-f6b1f279551a/volumes"
Sep 08 19:14:11.676568 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:11.676523    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:14:12.087007 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:12.086975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-p4lzw"
Sep 08 19:14:17.573095 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:17.573057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:14:24.575238 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:24.575161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qrpc7"]
Sep 08 19:14:33.264010 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:33.263981    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-9hdrt_6948f32a-4e7b-4ee9-bed1-398916c7995f/copy-manifests/0.log"
Sep 08 19:14:33.270453 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:33.270424    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-9hdrt_6948f32a-4e7b-4ee9-bed1-398916c7995f/manager/0.log"
Sep 08 19:14:33.812841 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:33.812809    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79cd597bc5-p4lzw_435ff9d4-9617-41fb-ba25-6cffe14e9843/maas-api/0.log"
Sep 08 19:14:35.097180 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:35.097150    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8njjp_f662d319-5f4b-4fae-80ed-02eae3098835/kuadrant-console-plugin/0.log"
Sep 08 19:14:35.344152 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:35.344125    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-qrpc7_1a37071b-4855-4b9a-be96-ec1899815bbc/limitador/0.log"
Sep 08 19:14:36.018297 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:36.018264    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54b8878898-fz9t7_9a4ebde5-218d-466c-91fe-2e68cc635bf7/kube-auth-proxy/0.log"
Sep 08 19:14:36.587823 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:36.587791    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cb4f5599b-zzlp9_07b49476-d0eb-4480-a6bf-314f4417473d/router/0.log"
Sep 08 19:14:36.923168 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:36.923074    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-c2v9j_dd90c2ea-59cb-4c8f-b921-ec593ffd0ffc/main/0.log"
Sep 08 19:14:37.368037 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:37.368012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcrl28_be70e0d4-c0ef-415b-b234-a2f24546b4ac/main/0.log"
Sep 08 19:14:37.597098 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:37.597060    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-96p4j_619602d0-e639-41c9-8a5e-bc2cca813481/main/0.log"
Sep 08 19:14:43.041429 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:43.041388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:14:43.140467 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:43.140424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log" limit=1024
Sep 08 19:14:45.854962 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:45.854930    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t5hjx_e680fcd6-db0f-49ff-88c8-94c448785e49/global-pull-secret-syncer/0.log"
Sep 08 19:14:45.939209 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:45.939179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bqqz7_24f743c5-c5d6-47ce-8e49-1865602059a9/konnectivity-agent/0.log"
Sep 08 19:14:46.071057 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:46.070951    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-26.ec2.internal_e01954d7983c7d343489f8e29f686ea0/haproxy/0.log"
Sep 08 19:14:50.852381 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:50.852328    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8njjp_f662d319-5f4b-4fae-80ed-02eae3098835/kuadrant-console-plugin/0.log"
Sep 08 19:14:50.922621 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:50.922592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-qrpc7_1a37071b-4855-4b9a-be96-ec1899815bbc/limitador/0.log"
Sep 08 19:14:52.406587 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.406555    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_14839ffa-94d7-4075-a2dd-85c53d4b5b28/alertmanager/0.log"
Sep 08 19:14:52.435046 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.435020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_14839ffa-94d7-4075-a2dd-85c53d4b5b28/config-reloader/0.log"
Sep 08 19:14:52.458082 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.458058    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_14839ffa-94d7-4075-a2dd-85c53d4b5b28/kube-rbac-proxy-web/0.log"
Sep 08 19:14:52.477709 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.477686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_14839ffa-94d7-4075-a2dd-85c53d4b5b28/kube-rbac-proxy/0.log"
Sep 08 19:14:52.496372 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.496294    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_14839ffa-94d7-4075-a2dd-85c53d4b5b28/kube-rbac-proxy-metric/0.log"
Sep 08 19:14:52.514131 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.514111    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_14839ffa-94d7-4075-a2dd-85c53d4b5b28/prom-label-proxy/0.log"
Sep 08 19:14:52.535227 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.535208    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_14839ffa-94d7-4075-a2dd-85c53d4b5b28/init-config-reloader/0.log"
Sep 08 19:14:52.967711 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.967673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jx7ch_3740c39d-855a-4039-98b5-c410709e9e9e/node-exporter/0.log"
Sep 08 19:14:52.990920 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:52.990898    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jx7ch_3740c39d-855a-4039-98b5-c410709e9e9e/kube-rbac-proxy/0.log"
Sep 08 19:14:53.014479 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.014455    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jx7ch_3740c39d-855a-4039-98b5-c410709e9e9e/init-textfile/0.log"
Sep 08 19:14:53.145432 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.145353    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2592e0d-d3ff-499f-ac51-8321e602937f/prometheus/0.log"
Sep 08 19:14:53.167512 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.167485    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2592e0d-d3ff-499f-ac51-8321e602937f/config-reloader/0.log"
Sep 08 19:14:53.186940 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.186915    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2592e0d-d3ff-499f-ac51-8321e602937f/thanos-sidecar/0.log"
Sep 08 19:14:53.214669 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.214634    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2592e0d-d3ff-499f-ac51-8321e602937f/kube-rbac-proxy-web/0.log"
Sep 08 19:14:53.234396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.234370    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2592e0d-d3ff-499f-ac51-8321e602937f/kube-rbac-proxy/0.log"
Sep 08 19:14:53.254592 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.254555    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2592e0d-d3ff-499f-ac51-8321e602937f/kube-rbac-proxy-thanos/0.log"
Sep 08 19:14:53.275179 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.275147    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2592e0d-d3ff-499f-ac51-8321e602937f/init-config-reloader/0.log"
Sep 08 19:14:53.315416 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.315387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-j496w_6d2bb26d-6a47-42d9-84be-14b2bfe31a54/prometheus-operator/0.log"
Sep 08 19:14:53.333890 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.333861    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-j496w_6d2bb26d-6a47-42d9-84be-14b2bfe31a54/kube-rbac-proxy/0.log"
Sep 08 19:14:53.357396 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.357371    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-qjptq_8b5dba47-b708-4ec5-8cb3-f006c08f9573/prometheus-operator-admission-webhook/0.log"
Sep 08 19:14:53.396050 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.395980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84455f89f4-9j82w_349ece67-7975-47ee-bc5c-7af76697caed/telemeter-client/0.log"
Sep 08 19:14:53.421018 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.420993    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84455f89f4-9j82w_349ece67-7975-47ee-bc5c-7af76697caed/reload/0.log"
Sep 08 19:14:53.438460 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:53.438436    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84455f89f4-9j82w_349ece67-7975-47ee-bc5c-7af76697caed/kube-rbac-proxy/0.log"
Sep 08 19:14:54.949556 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:54.949527    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-x6cmn_b0942923-36ad-4be6-afe4-b5c44d783a1c/networking-console-plugin/0.log"
Sep 08 19:14:55.776283 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.776240    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"]
Sep 08 19:14:55.776741 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.776720    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ca210a-7b21-4499-9c17-f6b1f279551a" containerName="maas-api"
Sep 08 19:14:55.776741 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.776743    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ca210a-7b21-4499-9c17-f6b1f279551a" containerName="maas-api"
Sep 08 19:14:55.776904 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.776865    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ca210a-7b21-4499-9c17-f6b1f279551a" containerName="maas-api"
Sep 08 19:14:55.779770 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.779747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.781926 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.781909    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qgbgk\"/\"kube-root-ca.crt\""
Sep 08 19:14:55.782468 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.782449    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qgbgk\"/\"default-dockercfg-xsjvx\""
Sep 08 19:14:55.782598 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.782449    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qgbgk\"/\"openshift-service-ca.crt\""
Sep 08 19:14:55.788688 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.788666    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"]
Sep 08 19:14:55.869118 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.869072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-sys\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.869293 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.869171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqrv\" (UniqueName: \"kubernetes.io/projected/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-kube-api-access-7xqrv\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.869293 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.869258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-proc\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.869407 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.869296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-lib-modules\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.869407 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.869322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-podres\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970186 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-podres\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970186 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-sys\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqrv\" (UniqueName: \"kubernetes.io/projected/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-kube-api-access-7xqrv\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-proc\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-lib-modules\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-podres\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-sys\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-proc\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.970618 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.970447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-lib-modules\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:55.977430 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:55.977410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqrv\" (UniqueName: \"kubernetes.io/projected/e311a692-fcf1-43b9-8899-5d6a9f5a1f05-kube-api-access-7xqrv\") pod \"perf-node-gather-daemonset-22tpm\" (UID: \"e311a692-fcf1-43b9-8899-5d6a9f5a1f05\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:56.089889 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:56.089861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:56.360035 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:56.359952    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"]
Sep 08 19:14:56.363188 ip-10-0-140-26 kubenswrapper[2579]: W0908 19:14:56.363163    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode311a692_fcf1_43b9_8899_5d6a9f5a1f05.slice/crio-af2e8624796898f92be202c548eb59a5b4943ebd461debc413e61ca289280a1d WatchSource:0}: Error finding container af2e8624796898f92be202c548eb59a5b4943ebd461debc413e61ca289280a1d: Status 404 returned error can't find the container with id af2e8624796898f92be202c548eb59a5b4943ebd461debc413e61ca289280a1d
Sep 08 19:14:57.311134 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:57.311098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm" event={"ID":"e311a692-fcf1-43b9-8899-5d6a9f5a1f05","Type":"ContainerStarted","Data":"5231481598e7ade4d61aaa36bf4c60e299590fbfbf1d71017e973cd78036f2aa"}
Sep 08 19:14:57.311523 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:57.311161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm" event={"ID":"e311a692-fcf1-43b9-8899-5d6a9f5a1f05","Type":"ContainerStarted","Data":"af2e8624796898f92be202c548eb59a5b4943ebd461debc413e61ca289280a1d"}
Sep 08 19:14:57.311523 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:57.311201    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:14:57.326012 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:57.325960    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm" podStartSLOduration=2.325943113 podStartE2EDuration="2.325943113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 19:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:14:57.32476857 +0000 UTC m=+2457.678861903" watchObservedRunningTime="2026-09-08 19:14:57.325943113 +0000 UTC m=+2457.680036448"
Sep 08 19:14:57.929833 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:57.929803    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nqt6r_c1654ffb-c4d0-421b-b162-28c6f1377f68/dns/0.log"
Sep 08 19:14:57.950427 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:57.950393    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nqt6r_c1654ffb-c4d0-421b-b162-28c6f1377f68/kube-rbac-proxy/0.log"
Sep 08 19:14:58.083826 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:58.083794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rn2xb_eb44a0dc-533c-4750-91aa-766b956e2465/dns-node-resolver/0.log"
Sep 08 19:14:58.761084 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:58.761056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kslr8_21b4eada-cc2e-4655-90b8-38e6f9bbcf6a/node-ca/0.log"
Sep 08 19:14:59.900381 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:14:59.900357    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54b8878898-fz9t7_9a4ebde5-218d-466c-91fe-2e68cc635bf7/kube-auth-proxy/0.log"
Sep 08 19:15:00.080281 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:00.080220    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cb4f5599b-zzlp9_07b49476-d0eb-4480-a6bf-314f4417473d/router/0.log"
Sep 08 19:15:00.928583 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:00.928552    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-flcd8_e2f54353-feeb-4c43-8b29-9c863b8dd040/serve-healthcheck-canary/0.log"
Sep 08 19:15:01.392724 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:01.392699    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-92rz7_1c0dc7ea-4220-4dc8-96e8-b7b534ce098f/kube-rbac-proxy/0.log"
Sep 08 19:15:01.410579 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:01.410554    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-92rz7_1c0dc7ea-4220-4dc8-96e8-b7b534ce098f/exporter/0.log"
Sep 08 19:15:01.428802 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:01.428779    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-92rz7_1c0dc7ea-4220-4dc8-96e8-b7b534ce098f/extractor/0.log"
Sep 08 19:15:03.326170 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:03.326136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-22tpm"
Sep 08 19:15:04.240483 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:04.240442    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-9hdrt_6948f32a-4e7b-4ee9-bed1-398916c7995f/manager/0.log"
Sep 08 19:15:04.259952 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:04.259916    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-9hdrt_6948f32a-4e7b-4ee9-bed1-398916c7995f/copy-manifests/0.log"
Sep 08 19:15:05.707926 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:05.707892    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68bb56fbd7-f5m5j_55216b79-6258-4fc1-8568-4d7f6559d00c/manager/0.log"
Sep 08 19:15:06.460158 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:06.460123    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54b8878898-fz9t7_9a4ebde5-218d-466c-91fe-2e68cc635bf7/kube-auth-proxy/0.log"
Sep 08 19:15:06.610707 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:06.610681    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cb4f5599b-zzlp9_07b49476-d0eb-4480-a6bf-314f4417473d/router/0.log"
Sep 08 19:15:08.357817 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:08.357773    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54b8878898-fz9t7_9a4ebde5-218d-466c-91fe-2e68cc635bf7/kube-auth-proxy/0.log"
Sep 08 19:15:08.505672 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:08.505642    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cb4f5599b-zzlp9_07b49476-d0eb-4480-a6bf-314f4417473d/router/0.log"
Sep 08 19:15:12.507980 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:12.507953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86pss_c50318ae-5b05-435f-9ba8-d2552408cdca/kube-multus-additional-cni-plugins/0.log"
Sep 08 19:15:12.526732 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:12.526711    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86pss_c50318ae-5b05-435f-9ba8-d2552408cdca/egress-router-binary-copy/0.log"
Sep 08 19:15:12.545361 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:12.545320    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86pss_c50318ae-5b05-435f-9ba8-d2552408cdca/cni-plugins/0.log"
Sep 08 19:15:12.563085 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:12.563066    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86pss_c50318ae-5b05-435f-9ba8-d2552408cdca/bond-cni-plugin/0.log"
Sep 08 19:15:12.580077 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:12.580059    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86pss_c50318ae-5b05-435f-9ba8-d2552408cdca/routeoverride-cni/0.log"
Sep 08 19:15:12.597156 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:12.597135    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86pss_c50318ae-5b05-435f-9ba8-d2552408cdca/whereabouts-cni-bincopy/0.log"
Sep 08 19:15:12.615123 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:12.615106    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86pss_c50318ae-5b05-435f-9ba8-d2552408cdca/whereabouts-cni/0.log"
Sep 08 19:15:12.922509 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:12.922481    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bw929_f05e2e6d-161d-4996-a7c6-e060e84fd1d1/kube-multus/0.log"
Sep 08 19:15:13.030943 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:13.030905    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8mlg8_7a40264b-ad31-4698-926a-5d35143a916f/network-metrics-daemon/0.log"
Sep 08 19:15:13.046259 ip-10-0-140-26 kubenswrapper[2579]: I0908 19:15:13.046238    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8mlg8_7a40264b-ad31-4698-926a-5d35143a916f/kube-rbac-proxy/0.log"