Sep 09 03:03:35.342860 ip-10-0-143-52 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 03:03:35.342873 ip-10-0-143-52 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 03:03:35.342881 ip-10-0-143-52 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 03:03:35.343101 ip-10-0-143-52 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 03:03:45.437566 ip-10-0-143-52 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 03:03:45.437587 ip-10-0-143-52 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 657bda2299b04dd9845f333389b1148d --
Sep 09 03:05:49.975575 ip-10-0-143-52 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 03:05:50.342449 ip-10-0-143-52 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:50.342449 ip-10-0-143-52 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 03:05:50.342449 ip-10-0-143-52 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:50.342449 ip-10-0-143-52 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 03:05:50.342449 ip-10-0-143-52 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:50.344912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.344818    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 03:05:50.347755 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347738    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.347755 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347755    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347759    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347763    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347766    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347769    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347772    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347775    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347777    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347780    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347782    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347785    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347787    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347791    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347794    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347796    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347800    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347802    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347805    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347808    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.347815 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347810    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347812    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347816    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347819    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347821    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347824    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347827    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347830    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347833    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347835    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347838    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347840    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347842    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347845    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347848    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347852    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347856    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347859    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347862    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.348279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347865    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347868    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347870    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347873    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347877    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347881    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347883    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347888    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347892    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347895    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347906    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347909    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347912    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347914    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347917    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347921    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347924    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347926    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347929    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347931    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.348739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347934    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347937    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347939    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347942    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347944    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347947    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347949    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347951    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347954    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347956    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347959    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347961    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347964    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347966    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347969    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347971    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347973    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347976    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347978    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347981    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.349213 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347984    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347986    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347988    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347991    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347994    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347996    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.347998    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348002    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348004    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348007    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348010    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348013    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348016    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348018    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348020    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348023    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348025    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348028    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348031    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348033    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.349739 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348767    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348774    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348777    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348780    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348783    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348785    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348790    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348794    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348797    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348800    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348802    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348805    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348808    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348810    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348813    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348815    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348818    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348822    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348824    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.350243 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348828    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348831    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348834    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348836    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348839    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348841    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348845    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348849    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348851    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348854    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348857    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348859    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348862    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348864    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348867    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348870    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348872    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348875    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348877    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348880    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.350813 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348884    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348888    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348891    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348894    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348896    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348900    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348902    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348905    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348908    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348910    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348913    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348915    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348918    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348921    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348924    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348927    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348929    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348932    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348934    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348937    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.351328 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348939    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348942    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348944    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348947    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348950    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348952    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348954    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348957    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348960    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348962    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348965    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348967    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348970    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348972    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348975    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348977    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348980    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348982    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348986    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.351812 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348989    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348992    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348994    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.348997    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349000    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349002    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349005    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349007    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349009    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349012    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349014    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349016    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349019    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349021    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349024    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349027    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349029    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349032    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349034    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349037    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.352287 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349040    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349495    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349501    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349504    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349507    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349510    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349513    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349516    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349519    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349522    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349525    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349528    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349531    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349534    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349538    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349542    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349545    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349548    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349551    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.352782 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349554    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349557    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349559    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349562    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349564    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349567    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349569    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349572    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349574    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349577    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349579    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349582    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349584    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349587    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349590    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349592    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349596    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349600    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349603    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.353257 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349606    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349608    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349611    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349613    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349616    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349618    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349621    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349624    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349626    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349629    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349632    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349634    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349637    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349639    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349642    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349644    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349647    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349649    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349651    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349654    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.353723 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349656    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349659    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349663    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349665    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349667    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349670    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349675    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349678    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349682    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349685    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349687    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349690    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349693    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349696    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349699    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349701    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349704    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349706    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349709    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349712    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.354193 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349714    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349717    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349719    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349722    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349725    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349727    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349730    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349732    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349735    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349737    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349740    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349743    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349745    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349748    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349751    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349753    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349756    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349758    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349762    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349765    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.354773 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349767    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.349770    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349838    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349875    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349882    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349887    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349891    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349894    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349899    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349903    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349906    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349909    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349913    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349916    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349919    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349922    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349925    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349928    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349931    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349934    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349939    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349941    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349944    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349947    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 03:05:50.355272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349951    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349955    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349957    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349961    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349965    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349968    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349973    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349976    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349979    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349982    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349985    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349988    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349991    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349994    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.349997    2562 flags.go:64] FLAG: --enable-server="true"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350000    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350004    2562 flags.go:64] FLAG: --event-burst="100"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350007    2562 flags.go:64] FLAG: --event-qps="50"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350010    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350013    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350016    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350019    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350022    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350025    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350028    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 09 03:05:50.355856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350031    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350034    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350037    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350040    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350043    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350046    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350048    2562 flags.go:64] FLAG: --feature-gates=""
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350053    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350056    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350059    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350062    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350066    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350069    2562 flags.go:64] FLAG: --help="false"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350073    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350076    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350079    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350082    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350086    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350089    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350092    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350095    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350097    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350100    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 03:05:50.356491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350103    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350106    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350109    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350111    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350114    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350117    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350120    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350123    2562 flags.go:64] FLAG: --lock-file=""
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350125    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350128    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350131    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350137    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350140    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350143    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350145    2562 flags.go:64] FLAG: --logging-format="text"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350148    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350151    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350154    2562 flags.go:64] FLAG: --manifest-url=""
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350157    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350161    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350165    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350169    2562 flags.go:64] FLAG: --max-pods="110"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350174    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350177    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350179    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 03:05:50.357102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350182    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350185    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350188    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350191    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350198    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350201    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350204    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350207    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350210    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350216    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350219    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350223    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350226    2562 flags.go:64] FLAG: --port="10250"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350243    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350246    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-089f177b0415b56ec"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350250    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350253    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350256    2562 flags.go:64] FLAG: --register-node="true"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350259    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350263    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350266    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350269    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350272    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350275    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350278    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 03:05:50.357858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350281    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350284    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350287    2562 flags.go:64] FLAG: --runonce="false"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350291    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350295    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350298    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350301    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350304    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350307    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350310    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350313    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350315    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350318    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350321    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350324    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350327    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350330    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350332    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350337    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350340    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350343    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350347    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350349    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350352    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350355    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 03:05:50.358898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350358    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350360    2562 flags.go:64] FLAG: --v="2"
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350365    2562 flags.go:64] FLAG: --version="false"
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350368    2562 flags.go:64] FLAG: --vmodule=""
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350372    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.350375    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350503    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350507    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350510    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350513    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350518    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350522    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350525    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350527    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350530    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350533    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350536    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350539    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350541    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350544    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350547    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.359573 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350549    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350551    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350554    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350557    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350559    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350562    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350564    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350567    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350569    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350571    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350574    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350576    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350579    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350581    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350585    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350587    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350589    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350592    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350594    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350597    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.360091 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350599    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350603    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350607    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350610    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350613    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350615    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350618    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350620    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350623    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350626    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350629    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350632    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350634    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350637    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350639    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350642    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350644    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350647    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350650    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350652    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.360696 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350655    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350657    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350661    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350665    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350668    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350671    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350673    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350676    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350679    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350682    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350684    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350687    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350689    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350693    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350697    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350700    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350702    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350705    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350707    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.361199 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350709    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350712    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350715    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350717    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350720    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350722    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350727    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350730    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350733    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350736    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350739    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350741    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350744    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350747    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350751    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350753    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350756    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350759    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350761    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.361701 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350764    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350766    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350769    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350772    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350774    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.350777    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.351299    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.359982    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.359998    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.360019    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.360025    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360079    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360084    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360088    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.362198 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360091    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360095    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360098    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360101    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360103    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360106    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360109    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360112    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360114    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360117    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360119    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360122    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360124    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360127    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360129    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360132    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360135    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360138    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360140    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360143    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.362572 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360145    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360148    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360151    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360153    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360155    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360158    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360161    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360164    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360166    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360169    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360172    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360175    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360177    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360180    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360182    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360184    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360187    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360189    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360192    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360194    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.363068 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360197    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360199    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360202    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360204    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360206    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360209    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360211    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360214    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360217    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360220    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360223    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360225    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360249    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360255    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360259    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360263    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360268    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360273    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360277    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.363578 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360280    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360283    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360285    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360288    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360292    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360296    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360300    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360304    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360308    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360311    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360314    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360316    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360319    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360321    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360324    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360327    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360329    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360339    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360341    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360344    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.364036 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360346    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360349    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360352    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360354    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360357    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360360    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360362    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360365    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360369    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360372    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360376    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360379    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360382    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360385    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360398    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360400    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.364580 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360403    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.360408    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360524    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360528    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360531    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360534    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360537    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360540    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360543    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360545    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360548    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360550    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360552    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360555    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.364975 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360564    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360567    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360570    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360572    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360575    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360577    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360580    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360582    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360585    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360587    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360590    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360592    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360595    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360598    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360601    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360603    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360606    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360608    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360611    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360614    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.365341 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360616    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360619    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360621    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360624    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360626    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360629    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360632    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360634    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360636    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360639    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360641    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360644    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360646    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360650    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360652    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360654    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360657    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360659    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360662    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360664    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.365817 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360667    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360669    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360672    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360674    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360676    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360679    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360682    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360684    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360688    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360691    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360694    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360696    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360699    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360702    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360705    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360707    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360710    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360714    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360716    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.366314 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360719    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360722    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360725    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360728    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360731    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360734    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360737    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360740    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360743    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360745    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360748    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360750    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360752    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360756    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360760    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360763    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360765    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360768    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360771    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.366764 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360774    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360776    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360779    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360781    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360784    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360786    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360788    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360791    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:50.360794    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.360798    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:50.367204 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.360913    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 03:05:50.367572 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.367557    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 03:05:50.368358 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.368345    2562 server.go:1019] "Starting client certificate rotation"
Sep 09 03:05:50.368464 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.368449    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:05:50.368500 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.368492    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:05:50.370605 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.370586    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:05:50.371753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.371739    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:05:50.381296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.381271    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 09 03:05:50.383981 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.383964    2562 log.go:25] "Validated CRI v1 image API"
Sep 09 03:05:50.384932 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.384916    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 03:05:50.390576 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.390556    2562 fs.go:135] Filesystem UUIDs: map[2c6adb6e-ac12-4a64-ba90-8001e6dd931a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a26f5226-8bba-4b4c-a1eb-4a357110825e:/dev/nvme0n1p3]
Sep 09 03:05:50.390634 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.390576    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 03:05:50.398217 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.397161    2562 manager.go:216] Machine: {Timestamp:2026-09-09 03:05:50.39560725 +0000 UTC m=+0.307239890 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100120 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246e86a245f9bd009874d678aafd9f SystemUUID:ec246e86-a245-f9bd-0098-74d678aafd9f BootID:657bda22-99b0-4dd9-845f-333389b1148d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:83:c4:11:ee:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:83:c4:11:ee:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:59:a5:1e:f0:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 03:05:50.398217 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.398213    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 03:05:50.398403 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.398368    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 03:05:50.401568 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.401539    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 03:05:50.401710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.401571    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 03:05:50.402200 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.402191    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 03:05:50.402229 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.402203    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 03:05:50.402229 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.402211    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:05:50.402866 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.402854    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:05:50.404004 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.403994    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 03:05:50.405415 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.405397    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:05:50.406704 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.406693    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:05:50.406843 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.406833    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 03:05:50.406873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.406864    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 03:05:50.406904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.406876    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 03:05:50.406904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.406888    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 03:05:50.406904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.406897    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 09 03:05:50.406982 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.406905    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 03:05:50.407900 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.407885    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:05:50.407976 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.407912    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:05:50.412150 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.412130    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 03:05:50.414036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414021    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 03:05:50.414096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414052    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 03:05:50.414247 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414224    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 03:05:50.414282 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414259    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 03:05:50.414282 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414266    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 03:05:50.414282 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414272    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 03:05:50.414282 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414279    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 03:05:50.414282 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414283    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 03:05:50.414597 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414589    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 03:05:50.414625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414597    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 03:05:50.414625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414604    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 03:05:50.414625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414609    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 03:05:50.414625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414616    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 03:05:50.414625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.414623    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 03:05:50.415299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.415289    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 03:05:50.415299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.415298    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 03:05:50.418722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.418702    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 03:05:50.418802 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.418752    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 03:05:50.418802 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.418770    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 03:05:50.419309 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.419295    2562 server.go:1268] "Started kubelet"
Sep 09 03:05:50.419371 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.419351    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 03:05:50.419526 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.419413    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 03:05:50.419608 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.419588    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 03:05:50.420135 ip-10-0-143-52 systemd[1]: Started Kubernetes Kubelet.
Sep 09 03:05:50.420686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.420655    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 03:05:50.424228 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.424151    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 03:05:50.426064 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.426010    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 03:05:50.427061 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.426963    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wv2rw"
Sep 09 03:05:50.428583 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.426141    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-52.ec2.internal.18d388208f3690df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-52.ec2.internal,UID:ip-10-0-143-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-52.ec2.internal,},FirstTimestamp:2026-09-09 03:05:50.419267807 +0000 UTC m=+0.330900446,LastTimestamp:2026-09-09 03:05:50.419267807 +0000 UTC m=+0.330900446,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-52.ec2.internal,}"
Sep 09 03:05:50.432993 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.432970    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 03:05:50.433303 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.433285    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wv2rw"
Sep 09 03:05:50.433699 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.433681    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 03:05:50.433979 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.433951    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:50.434478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.434459    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 03:05:50.434478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.434461    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 03:05:50.434627 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.434496    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 03:05:50.434682 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.434634    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 03:05:50.434682 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.434643    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 03:05:50.434771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.434754    2562 factory.go:55] Registering systemd factory
Sep 09 03:05:50.434771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.434767    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 09 03:05:50.434992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.434976    2562 factory.go:153] Registering CRI-O factory
Sep 09 03:05:50.435098 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.435085    2562 factory.go:223] Registration of the crio container factory successfully
Sep 09 03:05:50.435173 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.435163    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 03:05:50.435225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.435190    2562 factory.go:103] Registering Raw factory
Sep 09 03:05:50.435225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.435213    2562 manager.go:1196] Started watching for new ooms in manager
Sep 09 03:05:50.435344 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.435283    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 03:05:50.435574 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.435551    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 03:05:50.436208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.436189    2562 manager.go:349] Starting recovery of all containers
Sep 09 03:05:50.447757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.447739    2562 manager.go:354] Recovery completed
Sep 09 03:05:50.452257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.452226    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.454466 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.454447    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.454544 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.454479    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.454544 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.454490    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.455074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.455060    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 03:05:50.455074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.455073    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 03:05:50.455163 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.455093    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:05:50.455163 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.455096    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.456152 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.456140    2562 policy_none.go:49] "None policy: Start"
Sep 09 03:05:50.456209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.456155    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 03:05:50.456209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.456164    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 03:05:50.456617 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.456603    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.456658 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.456630    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.456658 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.456640    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.457154 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.457142    2562 policy_none.go:47] "Start"
Sep 09 03:05:50.499054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.499031    2562 manager.go:337] "Starting Device Plugin manager"
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.499084    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.499098    2562 server.go:87] "Starting device plugin registration server"
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.499514    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.499526    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.499616    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.499707    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.499721    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.500499    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 03:05:50.504784 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.500555    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:50.580658 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.580615    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 03:05:50.582017 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.581997    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 03:05:50.582079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.582068    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 03:05:50.582186 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.582097    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 03:05:50.582186 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.582138    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 03:05:50.585172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.585149    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:50.600203 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.600137    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.601139 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.601122    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.601198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.601155    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.601198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.601165    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.601198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.601188    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.609357 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.609336    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.609464 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.609366    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-52.ec2.internal\": node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:50.627363 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.627340    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:50.682619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.682592    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"]
Sep 09 03:05:50.682758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.682689    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.683794 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.683775    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.683914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.683804    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.683914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.683813    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.685102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685088    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.685271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685256    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.685336 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685295    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.685859 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685832    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.685930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685865    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.685930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685876    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.685930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685892    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.685930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685901    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.685930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.685879    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.687028 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.687013    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.687102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.687038    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.689201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.689161    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.689201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.689198    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.689347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.689211    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.717379 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.717348    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-52.ec2.internal\" not found" node="ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.722800 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.722771    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-52.ec2.internal\" not found" node="ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.728051 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.728028    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:50.828163 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.828105    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:50.835433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.835413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79fb75658caf768e20e5feb7474a2928-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"79fb75658caf768e20e5feb7474a2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.835503 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.835462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79fb75658caf768e20e5feb7474a2928-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"79fb75658caf768e20e5feb7474a2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.835554 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.835533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6639989942a5da5f614950f4219a4b4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-52.ec2.internal\" (UID: \"c6639989942a5da5f614950f4219a4b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.928925 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:50.928838    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:50.936133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.936110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79fb75658caf768e20e5feb7474a2928-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"79fb75658caf768e20e5feb7474a2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.936195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.936139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79fb75658caf768e20e5feb7474a2928-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"79fb75658caf768e20e5feb7474a2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.936195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.936173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6639989942a5da5f614950f4219a4b4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-52.ec2.internal\" (UID: \"c6639989942a5da5f614950f4219a4b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.936277 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.936223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79fb75658caf768e20e5feb7474a2928-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"79fb75658caf768e20e5feb7474a2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.936277 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.936257    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79fb75658caf768e20e5feb7474a2928-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"79fb75658caf768e20e5feb7474a2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:50.936338 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:50.936249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6639989942a5da5f614950f4219a4b4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-52.ec2.internal\" (UID: \"c6639989942a5da5f614950f4219a4b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:51.019292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.019227    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:51.025884 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.025866    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:51.029669 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.029653    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:51.130208 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.130167    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:51.230769 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.230731    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:51.331303 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.331253    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found"
Sep 09 03:05:51.357477 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.357442    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:51.368402 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.368365    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 03:05:51.368542 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.368522    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:05:51.368600 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.368572    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:05:51.402374 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.402345    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:51.408028 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.408008    2562 apiserver.go:52] "Watching apiserver"
Sep 09 03:05:51.416510 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.416479    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 03:05:51.416829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.416804    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-q765z","openshift-dns/node-resolver-lplwr","openshift-image-registry/node-ca-nwtvs","openshift-multus/multus-xr4dr","openshift-multus/network-metrics-daemon-w75g9","openshift-ovn-kubernetes/ovnkube-node-gdlm6","kube-system/global-pull-secret-syncer-4drfs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf","openshift-cluster-node-tuning-operator/tuned-nhr49","openshift-multus/multus-additional-cni-plugins-bgn7l","openshift-network-diagnostics/network-check-target-ccq7z","openshift-network-operator/iptables-alerter-pj78s"]
Sep 09 03:05:51.418207 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.418185    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:05:51.419392 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.419373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.420194 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.420177    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 03:05:51.420378 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.420221    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5qp5q\""
Sep 09 03:05:51.420378 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.420224    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 03:05:51.420541 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.420526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:51.420618 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.420600    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:05:51.421201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.421183    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 03:05:51.421507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.421476    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 03:05:51.421507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.421489    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5w9fz\""
Sep 09 03:05:51.421647 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.421540    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 03:05:51.421807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.421785    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.423472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.423454    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 03:05:51.423472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.423459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.423912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.423893    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 03:05:51.423912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.423905    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 03:05:51.424055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.423893    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 03:05:51.424055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.423917    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 03:05:51.424055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.423956    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 03:05:51.424055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.424033    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbgq6\""
Sep 09 03:05:51.424851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.424837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.425346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.425329    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 03:05:51.425436 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.425374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 03:05:51.425613 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.425594    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l9htr\""
Sep 09 03:05:51.425961 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.425950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.426033 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.426015    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:05:51.426479 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.426462    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 03:05:51.426522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.426489    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 03:05:51.427502 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.427061    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 03:05:51.427502 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.427309    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 03:05:51.427502 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.427434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.427636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.427533    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l87bb\""
Sep 09 03:05:51.429558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.429526    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 03:05:51.430104 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.430087    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 03:05:51.430208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.430140    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 03:05:51.430208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.430152    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-867vf\""
Sep 09 03:05:51.430345 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.430207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.431941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.431823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.432033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.431963    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xhrgh\""
Sep 09 03:05:51.432322 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.432204    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:05:51.432416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.432209    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 03:05:51.433097 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.433077    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 03:05:51.433714 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.433674    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hmw4t\""
Sep 09 03:05:51.433809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.433757    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 03:05:51.433893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.433877    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 03:05:51.434019 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.433990    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:51.434210 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.434191    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:51.434317 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.434293    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:05:51.435476 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.435418    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 03:00:51 +0000 UTC" deadline="2028-04-21 14:00:00.164465171 +0000 UTC"
Sep 09 03:05:51.435551 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.435476    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14170h54m8.72899306s"
Sep 09 03:05:51.435722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.435708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.437520 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.437500    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 03:05:51.437620 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.437567    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:05:51.437675 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.437624    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lchdn\""
Sep 09 03:05:51.437720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.437705    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 03:05:51.438935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.438919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysconfig\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.438991 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.438943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww9vh\" (UniqueName: \"kubernetes.io/projected/f2dbc178-5c81-4110-be6e-82b349c347fa-kube-api-access-ww9vh\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.438991 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.438961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74bd843b-133e-4ec2-a7a9-06b0461a7daa-iptables-alerter-script\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.438991 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.438983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-var-lib-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.439110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-k8s-cni-cncf-io\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.439110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-cni-multus\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.439110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-kubelet\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.439110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-hostroot\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.439276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/781959c0-527d-4192-86a4-99be89ec07ab-serviceca\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.439276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:51.439276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-systemd\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.439276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-log-socket\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.439276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-cni-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.439276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-daemon-config\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.439276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-multus-certs\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-lib-modules\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a66c6658-e660-435a-aec6-b3e64052cc35-ovn-node-metrics-cert\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5183151-84ae-456a-8341-d2df94d30764-tmp\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-cnibin\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-slash\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-node-log\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-cni-netd\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-ovnkube-script-lib\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439549    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqg6\" (UniqueName: \"kubernetes.io/projected/f66b535c-e240-4265-b350-8bed449f36f9-kube-api-access-4sqg6\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef19fc26-1f19-47bb-9d7d-3602e7c55f01-agent-certs\") pod \"konnectivity-agent-q765z\" (UID: \"ef19fc26-1f19-47bb-9d7d-3602e7c55f01\") " pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:05:51.439587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-env-overrides\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-sys\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-sys-fs\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74bd843b-133e-4ec2-a7a9-06b0461a7daa-host-slash\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcsr\" (UniqueName: \"kubernetes.io/projected/74bd843b-133e-4ec2-a7a9-06b0461a7daa-kube-api-access-kfcsr\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-etc-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57hv\" (UniqueName: \"kubernetes.io/projected/781959c0-527d-4192-86a4-99be89ec07ab-kube-api-access-w57hv\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-device-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef19fc26-1f19-47bb-9d7d-3602e7c55f01-konnectivity-ca\") pod \"konnectivity-agent-q765z\" (UID: \"ef19fc26-1f19-47bb-9d7d-3602e7c55f01\") " pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439823    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-kubelet\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/226d6682-de69-414a-bec1-1a3114c89844-hosts-file\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-dbus\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439929    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.440026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjc6c\" (UniqueName: \"kubernetes.io/projected/a66c6658-e660-435a-aec6-b3e64052cc35-kube-api-access-rjc6c\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.439984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-cni-bin\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-kubelet-config\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-systemd\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226d6682-de69-414a-bec1-1a3114c89844-tmp-dir\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-conf-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-var-lib-kubelet\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-host\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkdrc\" (UniqueName: \"kubernetes.io/projected/e5183151-84ae-456a-8341-d2df94d30764-kube-api-access-rkdrc\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-socket-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-systemd-units\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440497    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f799f4eb-9464-44af-aac1-ceb67e712d08-cni-binary-copy\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.440622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysctl-conf\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-run-netns\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-ovn\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-cni-bin\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-ovnkube-config\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440646    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p245s\" (UniqueName: \"kubernetes.io/projected/226d6682-de69-414a-bec1-1a3114c89844-kube-api-access-p245s\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-cnibin\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysctl-d\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5183151-84ae-456a-8341-d2df94d30764-etc-tuned\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-system-cni-dir\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/781959c0-527d-4192-86a4-99be89ec07ab-host\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs894\" (UniqueName: \"kubernetes.io/projected/c99b1fee-9619-4012-8bf1-19657cc1ac91-kube-api-access-hs894\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-registration-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-system-cni-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-netns\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-run\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.441276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-socket-dir-parent\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.441804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5lq\" (UniqueName: \"kubernetes.io/projected/f799f4eb-9464-44af-aac1-ceb67e712d08-kube-api-access-6h5lq\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.441804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.440995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-modprobe-d\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.441804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.441018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-kubernetes\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.441804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.441052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-etc-kubernetes\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.442143 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.442126    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:51.442198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.442193    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"
Sep 09 03:05:51.442268 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.442127    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"]
Sep 09 03:05:51.444043 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.444025    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:05:51.453295 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.453274    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:51.453445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.453428    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal"]
Sep 09 03:05:51.464325 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.464299    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wlgbt"
Sep 09 03:05:51.471197 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.471175    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wlgbt"
Sep 09 03:05:51.535013 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.534993    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 03:05:51.541849 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.541830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:51.541920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.541858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226d6682-de69-414a-bec1-1a3114c89844-tmp-dir\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.541920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.541876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-conf-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.541920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.541891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-var-lib-kubelet\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542057 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.541947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-conf-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.542057 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-host\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542057 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkdrc\" (UniqueName: \"kubernetes.io/projected/e5183151-84ae-456a-8341-d2df94d30764-kube-api-access-rkdrc\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.542192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-socket-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.542192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226d6682-de69-414a-bec1-1a3114c89844-tmp-dir\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.542192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-var-lib-kubelet\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542115    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-host\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-systemd-units\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-systemd-units\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f799f4eb-9464-44af-aac1-ceb67e712d08-cni-binary-copy\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysctl-conf\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-socket-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-run-netns\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-ovn\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-cni-bin\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542351    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-ovnkube-config\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-run-netns\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p245s\" (UniqueName: \"kubernetes.io/projected/226d6682-de69-414a-bec1-1a3114c89844-kube-api-access-p245s\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysctl-conf\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-cni-bin\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-cnibin\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-ovn\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.542447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysctl-d\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysctl-d\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5183151-84ae-456a-8341-d2df94d30764-etc-tuned\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-system-cni-dir\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/781959c0-527d-4192-86a4-99be89ec07ab-host\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs894\" (UniqueName: \"kubernetes.io/projected/c99b1fee-9619-4012-8bf1-19657cc1ac91-kube-api-access-hs894\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-registration-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-system-cni-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-netns\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f799f4eb-9464-44af-aac1-ceb67e712d08-cni-binary-copy\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542819    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/781959c0-527d-4192-86a4-99be89ec07ab-host\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-run\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.542886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542858    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-registration-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-system-cni-dir\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542906    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-system-cni-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-ovnkube-config\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-run\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-socket-dir-parent\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-cnibin\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-netns\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.542979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5lq\" (UniqueName: \"kubernetes.io/projected/f799f4eb-9464-44af-aac1-ceb67e712d08-kube-api-access-6h5lq\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-modprobe-d\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-socket-dir-parent\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543063    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-kubernetes\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-kubernetes\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-etc-kubernetes\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-modprobe-d\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysconfig\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.543296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-etc-kubernetes\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww9vh\" (UniqueName: \"kubernetes.io/projected/f2dbc178-5c81-4110-be6e-82b349c347fa-kube-api-access-ww9vh\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74bd843b-133e-4ec2-a7a9-06b0461a7daa-iptables-alerter-script\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-sysconfig\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543280    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-var-lib-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-k8s-cni-cncf-io\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-cni-multus\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-kubelet\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-var-lib-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-hostroot\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-hostroot\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/781959c0-527d-4192-86a4-99be89ec07ab-serviceca\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-k8s-cni-cncf-io\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-cni-multus\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-kubelet\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-systemd\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.544254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-log-socket\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-log-socket\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543656    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-systemd\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.543665    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-cni-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-daemon-config\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-multus-certs\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543758    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-run-multus-certs\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.543768    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:52.043728323 +0000 UTC m=+1.955360952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543806    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-cni-dir\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-lib-modules\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/781959c0-527d-4192-86a4-99be89ec07ab-serviceca\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a66c6658-e660-435a-aec6-b3e64052cc35-ovn-node-metrics-cert\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5183151-84ae-456a-8341-d2df94d30764-tmp\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-cnibin\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-slash\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.545110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-node-log\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543978    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-cni-netd\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-lib-modules\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.543998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-ovnkube-script-lib\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-slash\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2dbc178-5c81-4110-be6e-82b349c347fa-cnibin\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqg6\" (UniqueName: \"kubernetes.io/projected/f66b535c-e240-4265-b350-8bed449f36f9-kube-api-access-4sqg6\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef19fc26-1f19-47bb-9d7d-3602e7c55f01-agent-certs\") pod \"konnectivity-agent-q765z\" (UID: \"ef19fc26-1f19-47bb-9d7d-3602e7c55f01\") " pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544118    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-env-overrides\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-sys\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-sys-fs\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74bd843b-133e-4ec2-a7a9-06b0461a7daa-host-slash\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcsr\" (UniqueName: \"kubernetes.io/projected/74bd843b-133e-4ec2-a7a9-06b0461a7daa-kube-api-access-kfcsr\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-etc-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57hv\" (UniqueName: \"kubernetes.io/projected/781959c0-527d-4192-86a4-99be89ec07ab-kube-api-access-w57hv\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-device-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef19fc26-1f19-47bb-9d7d-3602e7c55f01-konnectivity-ca\") pod \"konnectivity-agent-q765z\" (UID: \"ef19fc26-1f19-47bb-9d7d-3602e7c55f01\") " pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-kubelet\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544481    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74bd843b-133e-4ec2-a7a9-06b0461a7daa-iptables-alerter-script\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/226d6682-de69-414a-bec1-1a3114c89844-hosts-file\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/226d6682-de69-414a-bec1-1a3114c89844-hosts-file\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f799f4eb-9464-44af-aac1-ceb67e712d08-multus-daemon-config\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-dbus\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-dbus\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-cni-netd\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjc6c\" (UniqueName: \"kubernetes.io/projected/a66c6658-e660-435a-aec6-b3e64052cc35-kube-api-access-rjc6c\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.544656    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-cni-bin\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-kubelet-config\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.546960 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.544712    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret podName:8a2c4bfb-6825-4173-93a0-96a27cbdaedd nodeName:}" failed. No retries permitted until 2026-09-09 03:05:52.044696284 +0000 UTC m=+1.956328928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret") pod "global-pull-secret-syncer-4drfs" (UID: "8a2c4bfb-6825-4173-93a0-96a27cbdaedd") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-kubelet-config\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-systemd\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-etc-systemd\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544821    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-etc-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-run-openvswitch\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-device-dir\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-ovnkube-script-lib\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545615    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5183151-84ae-456a-8341-d2df94d30764-sys\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef19fc26-1f19-47bb-9d7d-3602e7c55f01-konnectivity-ca\") pod \"konnectivity-agent-q765z\" (UID: \"ef19fc26-1f19-47bb-9d7d-3602e7c55f01\") " pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f66b535c-e240-4265-b350-8bed449f36f9-sys-fs\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.547784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-kubelet\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545736    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74bd843b-133e-4ec2-a7a9-06b0461a7daa-host-slash\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a66c6658-e660-435a-aec6-b3e64052cc35-env-overrides\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.544611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2dbc178-5c81-4110-be6e-82b349c347fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.545945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f799f4eb-9464-44af-aac1-ceb67e712d08-host-var-lib-cni-bin\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.546000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.546036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a66c6658-e660-435a-aec6-b3e64052cc35-node-log\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.546877    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.546898    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.546912    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gs8ns for pod openshift-network-diagnostics/network-check-target-ccq7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:51.546999    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns podName:2f0d4458-174a-415d-93cb-eb5fe526d927 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:52.04698624 +0000 UTC m=+1.958618885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gs8ns" (UniqueName: "kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns") pod "network-check-target-ccq7z" (UID: "2f0d4458-174a-415d-93cb-eb5fe526d927") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.547471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5183151-84ae-456a-8341-d2df94d30764-tmp\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.547512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5183151-84ae-456a-8341-d2df94d30764-etc-tuned\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.547535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a66c6658-e660-435a-aec6-b3e64052cc35-ovn-node-metrics-cert\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.548865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.548348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef19fc26-1f19-47bb-9d7d-3602e7c55f01-agent-certs\") pod \"konnectivity-agent-q765z\" (UID: \"ef19fc26-1f19-47bb-9d7d-3602e7c55f01\") " pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:05:51.549756 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.549736    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p245s\" (UniqueName: \"kubernetes.io/projected/226d6682-de69-414a-bec1-1a3114c89844-kube-api-access-p245s\") pod \"node-resolver-lplwr\" (UID: \"226d6682-de69-414a-bec1-1a3114c89844\") " pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:51.550059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.550037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkdrc\" (UniqueName: \"kubernetes.io/projected/e5183151-84ae-456a-8341-d2df94d30764-kube-api-access-rkdrc\") pod \"tuned-nhr49\" (UID: \"e5183151-84ae-456a-8341-d2df94d30764\") " pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.553663 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:51.552434    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fb75658caf768e20e5feb7474a2928.slice/crio-991fd40bb333636909de22d7ae1077d3343ea00f98e843ef11e45ef5761eab04 WatchSource:0}: Error finding container 991fd40bb333636909de22d7ae1077d3343ea00f98e843ef11e45ef5761eab04: Status 404 returned error can't find the container with id 991fd40bb333636909de22d7ae1077d3343ea00f98e843ef11e45ef5761eab04
Sep 09 03:05:51.554026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.553990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5lq\" (UniqueName: \"kubernetes.io/projected/f799f4eb-9464-44af-aac1-ceb67e712d08-kube-api-access-6h5lq\") pod \"multus-xr4dr\" (UID: \"f799f4eb-9464-44af-aac1-ceb67e712d08\") " pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:51.554148 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.554124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs894\" (UniqueName: \"kubernetes.io/projected/c99b1fee-9619-4012-8bf1-19657cc1ac91-kube-api-access-hs894\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:51.554445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.554384    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww9vh\" (UniqueName: \"kubernetes.io/projected/f2dbc178-5c81-4110-be6e-82b349c347fa-kube-api-access-ww9vh\") pod \"multus-additional-cni-plugins-bgn7l\" (UID: \"f2dbc178-5c81-4110-be6e-82b349c347fa\") " pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.555130 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.555104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57hv\" (UniqueName: \"kubernetes.io/projected/781959c0-527d-4192-86a4-99be89ec07ab-kube-api-access-w57hv\") pod \"node-ca-nwtvs\" (UID: \"781959c0-527d-4192-86a4-99be89ec07ab\") " pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:51.556104 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.556085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqg6\" (UniqueName: \"kubernetes.io/projected/f66b535c-e240-4265-b350-8bed449f36f9-kube-api-access-4sqg6\") pod \"aws-ebs-csi-driver-node-2frdf\" (UID: \"f66b535c-e240-4265-b350-8bed449f36f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:51.556245 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.556146    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjc6c\" (UniqueName: \"kubernetes.io/projected/a66c6658-e660-435a-aec6-b3e64052cc35-kube-api-access-rjc6c\") pod \"ovnkube-node-gdlm6\" (UID: \"a66c6658-e660-435a-aec6-b3e64052cc35\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:51.556511 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.556492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcsr\" (UniqueName: \"kubernetes.io/projected/74bd843b-133e-4ec2-a7a9-06b0461a7daa-kube-api-access-kfcsr\") pod \"iptables-alerter-pj78s\" (UID: \"74bd843b-133e-4ec2-a7a9-06b0461a7daa\") " pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.558359 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.558342    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:05:51.561747 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.561733    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nhr49"
Sep 09 03:05:51.567188 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.567170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgn7l"
Sep 09 03:05:51.570749 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:51.570720    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5183151_84ae_456a_8341_d2df94d30764.slice/crio-cf86a5f3227252af215072726b408dc771230cedc06c78a290b59663ef81445e WatchSource:0}: Error finding container cf86a5f3227252af215072726b408dc771230cedc06c78a290b59663ef81445e: Status 404 returned error can't find the container with id cf86a5f3227252af215072726b408dc771230cedc06c78a290b59663ef81445e
Sep 09 03:05:51.574612 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.574589    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pj78s"
Sep 09 03:05:51.578298 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:51.578272    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dbc178_5c81_4110_be6e_82b349c347fa.slice/crio-f87f28244ae0ae6d260a765c67f439e070245483fec7cf1f21fa94bd21614357 WatchSource:0}: Error finding container f87f28244ae0ae6d260a765c67f439e070245483fec7cf1f21fa94bd21614357: Status 404 returned error can't find the container with id f87f28244ae0ae6d260a765c67f439e070245483fec7cf1f21fa94bd21614357
Sep 09 03:05:51.585451 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:51.585409    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bd843b_133e_4ec2_a7a9_06b0461a7daa.slice/crio-18ca2c527c8dc09925f1af33e500cd51534003dac6e94c9ef4e106eea672e966 WatchSource:0}: Error finding container 18ca2c527c8dc09925f1af33e500cd51534003dac6e94c9ef4e106eea672e966: Status 404 returned error can't find the container with id 18ca2c527c8dc09925f1af33e500cd51534003dac6e94c9ef4e106eea672e966
Sep 09 03:05:51.587005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.586955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nhr49" event={"ID":"e5183151-84ae-456a-8341-d2df94d30764","Type":"ContainerStarted","Data":"cf86a5f3227252af215072726b408dc771230cedc06c78a290b59663ef81445e"}
Sep 09 03:05:51.588695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.588668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" event={"ID":"79fb75658caf768e20e5feb7474a2928","Type":"ContainerStarted","Data":"991fd40bb333636909de22d7ae1077d3343ea00f98e843ef11e45ef5761eab04"}
Sep 09 03:05:51.589761 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:51.589738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" event={"ID":"f2dbc178-5c81-4110-be6e-82b349c347fa","Type":"ContainerStarted","Data":"f87f28244ae0ae6d260a765c67f439e070245483fec7cf1f21fa94bd21614357"}
Sep 09 03:05:53.099379 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.099324    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:05:53.107858 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:53.107831    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef19fc26_1f19_47bb_9d7d_3602e7c55f01.slice/crio-3a4672e9edf3e5a5a33e3509cf4d0e307f7ffd476168eeb1e5c39fe43d3bc236 WatchSource:0}: Error finding container 3a4672e9edf3e5a5a33e3509cf4d0e307f7ffd476168eeb1e5c39fe43d3bc236: Status 404 returned error can't find the container with id 3a4672e9edf3e5a5a33e3509cf4d0e307f7ffd476168eeb1e5c39fe43d3bc236
Sep 09 03:05:53.121418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.121394    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwtvs"
Sep 09 03:05:53.130095 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:53.130059    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781959c0_527d_4192_86a4_99be89ec07ab.slice/crio-659da95a13c1162093516fd5423c4909bc4c75bcf9157e14d6e07da8da51ef6d WatchSource:0}: Error finding container 659da95a13c1162093516fd5423c4909bc4c75bcf9157e14d6e07da8da51ef6d: Status 404 returned error can't find the container with id 659da95a13c1162093516fd5423c4909bc4c75bcf9157e14d6e07da8da51ef6d
Sep 09 03:05:53.145632 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.145603    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:05:53.153444 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:53.153419    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66c6658_e660_435a_aec6_b3e64052cc35.slice/crio-0907c347af1d9b2cdd434a313ef71eab9653d824b07cced0e0bb731d0cd1f32e WatchSource:0}: Error finding container 0907c347af1d9b2cdd434a313ef71eab9653d824b07cced0e0bb731d0cd1f32e: Status 404 returned error can't find the container with id 0907c347af1d9b2cdd434a313ef71eab9653d824b07cced0e0bb731d0cd1f32e
Sep 09 03:05:53.157433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.157414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lplwr"
Sep 09 03:05:53.164623 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:53.164601    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226d6682_de69_414a_bec1_1a3114c89844.slice/crio-5010710c8ab787f0ae4829be98015f70acda7928f425826cbe6f8131dc517f66 WatchSource:0}: Error finding container 5010710c8ab787f0ae4829be98015f70acda7928f425826cbe6f8131dc517f66: Status 404 returned error can't find the container with id 5010710c8ab787f0ae4829be98015f70acda7928f425826cbe6f8131dc517f66
Sep 09 03:05:53.179429 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.179393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xr4dr"
Sep 09 03:05:53.186765 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:53.186740    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf799f4eb_9464_44af_aac1_ceb67e712d08.slice/crio-4e193594756fc7959d1e693700447f6d50cb9a633e82c0631a747abea75135be WatchSource:0}: Error finding container 4e193594756fc7959d1e693700447f6d50cb9a633e82c0631a747abea75135be: Status 404 returned error can't find the container with id 4e193594756fc7959d1e693700447f6d50cb9a633e82c0631a747abea75135be
Sep 09 03:05:53.198446 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.198420    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf"
Sep 09 03:05:53.206460 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:05:53.206435    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66b535c_e240_4265_b350_8bed449f36f9.slice/crio-7dba90374cc07e7c4e11c44b661aa3c16e5deba6d098af97b1a7d98ca1dd7788 WatchSource:0}: Error finding container 7dba90374cc07e7c4e11c44b661aa3c16e5deba6d098af97b1a7d98ca1dd7788: Status 404 returned error can't find the container with id 7dba90374cc07e7c4e11c44b661aa3c16e5deba6d098af97b1a7d98ca1dd7788
Sep 09 03:05:53.340916 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.340864    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:53.399149 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.399071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:53.399149 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.399126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.399155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.399243    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.399243    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.399262    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.399276    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.399287    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gs8ns for pod openshift-network-diagnostics/network-check-target-ccq7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.399307    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret podName:8a2c4bfb-6825-4173-93a0-96a27cbdaedd nodeName:}" failed. No retries permitted until 2026-09-09 03:05:54.399287642 +0000 UTC m=+2.961614170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret") pod "global-pull-secret-syncer-4drfs" (UID: "8a2c4bfb-6825-4173-93a0-96a27cbdaedd") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.399324    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns podName:2f0d4458-174a-415d-93cb-eb5fe526d927 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:54.399312803 +0000 UTC m=+2.961639349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs8ns" (UniqueName: "kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns") pod "network-check-target-ccq7z" (UID: "2f0d4458-174a-415d-93cb-eb5fe526d927") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:53.399354 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.399342    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:54.399332987 +0000 UTC m=+2.961659511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:53.822835 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.822757    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:00:52 +0000 UTC" deadline="2028-02-28 02:16:46.979073313 +0000 UTC"
Sep 09 03:05:53.822835 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.822818    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12887h10m53.156259799s"
Sep 09 03:05:53.936228 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.934976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:53.936228 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.935096    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:05:53.936228 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.935701    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:53.936228 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:53.935824    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:05:53.950336 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.950298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lplwr" event={"ID":"226d6682-de69-414a-bec1-1a3114c89844","Type":"ContainerStarted","Data":"5010710c8ab787f0ae4829be98015f70acda7928f425826cbe6f8131dc517f66"}
Sep 09 03:05:53.958453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.958371    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"0907c347af1d9b2cdd434a313ef71eab9653d824b07cced0e0bb731d0cd1f32e"}
Sep 09 03:05:53.962196 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.962161    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pj78s" event={"ID":"74bd843b-133e-4ec2-a7a9-06b0461a7daa","Type":"ContainerStarted","Data":"18ca2c527c8dc09925f1af33e500cd51534003dac6e94c9ef4e106eea672e966"}
Sep 09 03:05:53.964355 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.964326    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf" event={"ID":"f66b535c-e240-4265-b350-8bed449f36f9","Type":"ContainerStarted","Data":"7dba90374cc07e7c4e11c44b661aa3c16e5deba6d098af97b1a7d98ca1dd7788"}
Sep 09 03:05:53.972469 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.972434    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xr4dr" event={"ID":"f799f4eb-9464-44af-aac1-ceb67e712d08","Type":"ContainerStarted","Data":"4e193594756fc7959d1e693700447f6d50cb9a633e82c0631a747abea75135be"}
Sep 09 03:05:53.979577 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.979505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwtvs" event={"ID":"781959c0-527d-4192-86a4-99be89ec07ab","Type":"ContainerStarted","Data":"659da95a13c1162093516fd5423c4909bc4c75bcf9157e14d6e07da8da51ef6d"}
Sep 09 03:05:53.983133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:53.983098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q765z" event={"ID":"ef19fc26-1f19-47bb-9d7d-3602e7c55f01","Type":"ContainerStarted","Data":"3a4672e9edf3e5a5a33e3509cf4d0e307f7ffd476168eeb1e5c39fe43d3bc236"}
Sep 09 03:05:54.180514 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:54.180412    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:54.392262 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:54.392231    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:54.408208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:54.408127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:54.408208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:54.408180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:54.408449 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:54.408230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:54.408449 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.408382    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:54.408449 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.408450    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:56.408429054 +0000 UTC m=+4.970755578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:54.408932 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.408910    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:54.409027 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.408968    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret podName:8a2c4bfb-6825-4173-93a0-96a27cbdaedd nodeName:}" failed. No retries permitted until 2026-09-09 03:05:56.408953196 +0000 UTC m=+4.971279725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret") pod "global-pull-secret-syncer-4drfs" (UID: "8a2c4bfb-6825-4173-93a0-96a27cbdaedd") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:54.409086 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.409050    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:54.409086 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.409064    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:54.409086 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.409076    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gs8ns for pod openshift-network-diagnostics/network-check-target-ccq7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:54.409229 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.409112    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns podName:2f0d4458-174a-415d-93cb-eb5fe526d927 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:56.409099891 +0000 UTC m=+4.971426421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs8ns" (UniqueName: "kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns") pod "network-check-target-ccq7z" (UID: "2f0d4458-174a-415d-93cb-eb5fe526d927") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:54.823333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:54.823274    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:00:52 +0000 UTC" deadline="2028-02-26 03:27:54.397463221 +0000 UTC"
Sep 09 03:05:54.823333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:54.823320    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12840h21m59.574147799s"
Sep 09 03:05:54.931981 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:54.931935    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:54.932171 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:54.932090    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:05:55.931977 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:55.931947    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:55.932611 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:55.932105    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:05:55.937903 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:55.937609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:55.937903 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:55.937825    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:05:56.428456 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:56.428277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:56.428456 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:56.428335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:56.428456 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:56.428392    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:56.428777 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.428529    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:56.428777 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.428611    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:00.428590433 +0000 UTC m=+8.990916957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:56.428777 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.428666    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:56.428777 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.428684    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:56.428777 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.428703    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:56.428777 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.428713    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gs8ns for pod openshift-network-diagnostics/network-check-target-ccq7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:56.428777 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.428736    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret podName:8a2c4bfb-6825-4173-93a0-96a27cbdaedd nodeName:}" failed. No retries permitted until 2026-09-09 03:06:00.428710329 +0000 UTC m=+8.991036855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret") pod "global-pull-secret-syncer-4drfs" (UID: "8a2c4bfb-6825-4173-93a0-96a27cbdaedd") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:56.428777 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.428753    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns podName:2f0d4458-174a-415d-93cb-eb5fe526d927 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:00.428742937 +0000 UTC m=+8.991069465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs8ns" (UniqueName: "kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns") pod "network-check-target-ccq7z" (UID: "2f0d4458-174a-415d-93cb-eb5fe526d927") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:56.932440 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:56.931889    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:56.932440 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:56.932029    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:05:57.932381 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:57.932344    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:57.932590 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:57.932478    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:05:57.932978 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:57.932950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:57.933096 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:57.933071    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:05:58.932675 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:58.932641    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:05:58.933146 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:58.932773    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:05:59.932149 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:59.932116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:05:59.932337 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:59.932240    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:05:59.932850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:05:59.932705    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:05:59.932850 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:05:59.932818    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:00.461612 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:00.461570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:00.461797 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:00.461623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:00.461797 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:00.461652    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:00.461797 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.461757    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:00.461797 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.461769    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:00.461797 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.461777    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:00.461797 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.461800    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:00.462032 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.461811    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gs8ns for pod openshift-network-diagnostics/network-check-target-ccq7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:00.462032 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.461838    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:08.461813865 +0000 UTC m=+17.024140396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:00.462032 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.461858    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret podName:8a2c4bfb-6825-4173-93a0-96a27cbdaedd nodeName:}" failed. No retries permitted until 2026-09-09 03:06:08.461847953 +0000 UTC m=+17.024174477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret") pod "global-pull-secret-syncer-4drfs" (UID: "8a2c4bfb-6825-4173-93a0-96a27cbdaedd") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:00.462032 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.461871    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns podName:2f0d4458-174a-415d-93cb-eb5fe526d927 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:08.461865246 +0000 UTC m=+17.024191772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs8ns" (UniqueName: "kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns") pod "network-check-target-ccq7z" (UID: "2f0d4458-174a-415d-93cb-eb5fe526d927") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:00.932381 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:00.931972    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:00.932381 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:00.932120    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:01.933457 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:01.933419    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:01.933949 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:01.933529    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:01.933949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:01.933904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:01.934052 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:01.933986    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:02.932512 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:02.932473    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:02.932731 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:02.932638    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:03.932346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:03.932253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:03.932839 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:03.932261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:03.932839 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:03.932388    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:03.932839 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:03.932446    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:04.932444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:04.932413    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:04.932925 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:04.932531    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:05.932117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:05.932075    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:05.932320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:05.932092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:05.932320 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:05.932223    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:05.932433 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:05.932333    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:06.932082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:06.932047    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:06.932577 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:06.932160    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:07.931918 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:07.931881    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:07.932115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:07.931881    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:07.932115 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:07.932030    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:07.932484 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:07.932104    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:08.526885 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:08.526833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:08.526900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:08.526930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.527013    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.527044    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.527042    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.527058    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.527075    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gs8ns for pod openshift-network-diagnostics/network-check-target-ccq7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.527075    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:24.527057857 +0000 UTC m=+33.089384399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.527114    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret podName:8a2c4bfb-6825-4173-93a0-96a27cbdaedd nodeName:}" failed. No retries permitted until 2026-09-09 03:06:24.527104227 +0000 UTC m=+33.089430757 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret") pod "global-pull-secret-syncer-4drfs" (UID: "8a2c4bfb-6825-4173-93a0-96a27cbdaedd") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:08.527153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.527127    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns podName:2f0d4458-174a-415d-93cb-eb5fe526d927 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:24.527119141 +0000 UTC m=+33.089445664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs8ns" (UniqueName: "kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns") pod "network-check-target-ccq7z" (UID: "2f0d4458-174a-415d-93cb-eb5fe526d927") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:08.932520 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:08.932488    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:08.933028 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:08.932639    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:09.932530 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:09.932490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:09.932732 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:09.932665    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:09.932732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:09.932706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:09.933162 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:09.932779    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:10.931689 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:10.931660    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:10.931844 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:10.931753    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:11.933024 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:11.932758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:11.933467 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:11.932828    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:11.933467 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:11.933136    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:11.933467 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:11.933192    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:12.024383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:12.024128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"0debac576705ba93e7280c9f8d5452a09b40892870686309e01ef8f17cf04d36"}
Sep 09 03:06:12.025871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:12.025836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" event={"ID":"c6639989942a5da5f614950f4219a4b4","Type":"ContainerStarted","Data":"851ca4539d993ff444db7574d2ef124f9484f81894a9f8e16e718c6eb23e35a8"}
Sep 09 03:06:12.027944 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:12.027872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nhr49" event={"ID":"e5183151-84ae-456a-8341-d2df94d30764","Type":"ContainerStarted","Data":"2167ec2dd347c293faad6a2935253186e9b799983ac25be2b6563ed5048a8c42"}
Sep 09 03:06:12.029994 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:12.029950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xr4dr" event={"ID":"f799f4eb-9464-44af-aac1-ceb67e712d08","Type":"ContainerStarted","Data":"091fe7936a389d73bbcfe5032623b1ceae67651f565219eb2fd1fa7aa831110c"}
Sep 09 03:06:12.043365 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:12.043296    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nhr49" podStartSLOduration=1.777686331 podStartE2EDuration="20.043274091s" totalImagesPullingTime="18.26558776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:12.04216922 +0000 UTC m=+20.604495764" watchObservedRunningTime="2026-09-09 03:06:12.043274091 +0000 UTC m=+20.605600638"
Sep 09 03:06:12.057060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:12.056994    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xr4dr" podStartSLOduration=2.053228676 podStartE2EDuration="20.056970755s" totalImagesPullingTime="18.003742079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:12.056437812 +0000 UTC m=+20.618764358" watchObservedRunningTime="2026-09-09 03:06:12.056970755 +0000 UTC m=+20.619297306"
Sep 09 03:06:12.932342 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:12.932304    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:12.932532 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:12.932438    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:13.034929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.034889    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lplwr" event={"ID":"226d6682-de69-414a-bec1-1a3114c89844","Type":"ContainerStarted","Data":"d52ea44b133e0f4eea9ed16f02283836ecdf832f9e91350d37dc909d3e1353c0"}
Sep 09 03:06:13.038086 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.038060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"112998c0493612bd90fe582ba2139f279b4135c588ca0f879624561f796998d4"}
Sep 09 03:06:13.038219 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.038140    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"1b371ea429d6cd82691fc8eeb69986bada9243e4cc4d46daae360c277e7d166f"}
Sep 09 03:06:13.038219 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.038203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"3f95bff7c68d7d6afc47757d9168fd965bbcd4210723f96cb79a410651e30330"}
Sep 09 03:06:13.038325 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.038252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"45c4a810dc9d7ed983bd6b03491fa693bd582fca0002d5744990fc18bfe450e4"}
Sep 09 03:06:13.039607 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.039575    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pj78s" event={"ID":"74bd843b-133e-4ec2-a7a9-06b0461a7daa","Type":"ContainerStarted","Data":"d874c01309a30668460de2438d309af2c002d40c3e6fa62c4916eaf92c7075bb"}
Sep 09 03:06:13.041151 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.041126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" event={"ID":"c6639989942a5da5f614950f4219a4b4","Type":"ContainerStarted","Data":"b923fd32c026b2cb07b0918b63545430cff45781bc9bf2f638305d22444b91c2"}
Sep 09 03:06:13.042914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.042889    2562 generic.go:356] "Generic (PLEG): container finished" podID="79fb75658caf768e20e5feb7474a2928" containerID="0115574c4ec496b28a5591ab0f3bcd2eece0b7ff51e5182693c42e51d7834e19" exitCode=0
Sep 09 03:06:13.042999 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.042975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" event={"ID":"79fb75658caf768e20e5feb7474a2928","Type":"ContainerDied","Data":"0115574c4ec496b28a5591ab0f3bcd2eece0b7ff51e5182693c42e51d7834e19"}
Sep 09 03:06:13.044999 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.044972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf" event={"ID":"f66b535c-e240-4265-b350-8bed449f36f9","Type":"ContainerStarted","Data":"72d3c2d7014592f5999780d4fb2c156160ad6cc7bed229caee329d5153645db7"}
Sep 09 03:06:13.046681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.046625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwtvs" event={"ID":"781959c0-527d-4192-86a4-99be89ec07ab","Type":"ContainerStarted","Data":"ab222c9857597209908b743c7091298de7462c51a3d8210bc4bdb31416c659ff"}
Sep 09 03:06:13.047695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.047656    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lplwr" podStartSLOduration=6.8736737009999995 podStartE2EDuration="21.047644585s" totalImagesPullingTime="14.173970884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:13.047294466 +0000 UTC m=+21.609621010" watchObservedRunningTime="2026-09-09 03:06:13.047644585 +0000 UTC m=+21.609971128"
Sep 09 03:06:13.048338 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.048319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q765z" event={"ID":"ef19fc26-1f19-47bb-9d7d-3602e7c55f01","Type":"ContainerStarted","Data":"3c8f2e5b9ba90ed792042e748babc4947676ab319d3a58c067a1c985323eff67"}
Sep 09 03:06:13.050655 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.050636    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2dbc178-5c81-4110-be6e-82b349c347fa" containerID="beca2b40ec1f611ddb45f3c3ce06f86b0f195e4414a30e11ec16218fd5b52df0" exitCode=0
Sep 09 03:06:13.050863 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.050839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" event={"ID":"f2dbc178-5c81-4110-be6e-82b349c347fa","Type":"ContainerDied","Data":"beca2b40ec1f611ddb45f3c3ce06f86b0f195e4414a30e11ec16218fd5b52df0"}
Sep 09 03:06:13.061112 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.061071    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pj78s" podStartSLOduration=2.808448849 podStartE2EDuration="21.061049667s" totalImagesPullingTime="18.252600818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:13.060799475 +0000 UTC m=+21.623126021" watchObservedRunningTime="2026-09-09 03:06:13.061049667 +0000 UTC m=+21.623376213"
Sep 09 03:06:13.116938 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.116876    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" podStartSLOduration=21.116857719 podStartE2EDuration="21.116857719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:13.116697738 +0000 UTC m=+21.679024310" watchObservedRunningTime="2026-09-09 03:06:13.116857719 +0000 UTC m=+21.679184263"
Sep 09 03:06:13.130434 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.130338    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nwtvs" podStartSLOduration=4.093584472 podStartE2EDuration="22.130323479s" totalImagesPullingTime="18.036739007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:13.129886309 +0000 UTC m=+21.692212854" watchObservedRunningTime="2026-09-09 03:06:13.130323479 +0000 UTC m=+21.692650025"
Sep 09 03:06:13.165060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.164987    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q765z" podStartSLOduration=7.934385238 podStartE2EDuration="22.164965036s" totalImagesPullingTime="14.230579798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:13.164293413 +0000 UTC m=+21.726619958" watchObservedRunningTime="2026-09-09 03:06:13.164965036 +0000 UTC m=+21.727291581"
Sep 09 03:06:13.564827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.564601    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 03:06:13.861393 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.861229    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T03:06:13.56482349Z","UUID":"af198906-41a3-4e37-a915-f88af7a745ab","Handler":null,"Name":"","Endpoint":""}
Sep 09 03:06:13.863060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.863029    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 03:06:13.863060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.863063    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 03:06:13.931874 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.931836    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:13.932165 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:13.931990    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:13.932165 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:13.932035    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:13.932165 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:13.932153    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:14.055329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:14.055289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" event={"ID":"79fb75658caf768e20e5feb7474a2928","Type":"ContainerStarted","Data":"00f03e2d3b82fb529a1c9be72f49fa126236d691348444dfa113d21b01637136"}
Sep 09 03:06:14.057556 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:14.057514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf" event={"ID":"f66b535c-e240-4265-b350-8bed449f36f9","Type":"ContainerStarted","Data":"9f0bd6f0d1c74df6af72c6bf98d24a7e5674bde28953dcb6f23b6a54bfb906aa"}
Sep 09 03:06:14.061112 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:14.061081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"13d414aed7b19e48ad59abcfb5812aa480efc7940d061ce3791d8ee459f39bf5"}
Sep 09 03:06:14.068001 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:14.067945    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" podStartSLOduration=22.06792955 podStartE2EDuration="22.06792955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:14.067697344 +0000 UTC m=+22.630023907" watchObservedRunningTime="2026-09-09 03:06:14.06792955 +0000 UTC m=+22.630256094"
Sep 09 03:06:14.932205 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:14.932126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:14.932350 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:14.932229    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:15.066909 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:15.066873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf" event={"ID":"f66b535c-e240-4265-b350-8bed449f36f9","Type":"ContainerStarted","Data":"19e5e1df4eff925bcfde800e59625c4295bd5c2651a9e4f08de7c4a0a293d350"}
Sep 09 03:06:15.082715 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:15.082662    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2frdf" podStartSLOduration=6.672128366 podStartE2EDuration="23.082644903s" totalImagesPullingTime="16.410516537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:15.082382494 +0000 UTC m=+23.644709041" watchObservedRunningTime="2026-09-09 03:06:15.082644903 +0000 UTC m=+23.644971448"
Sep 09 03:06:15.932078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:15.932044    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:15.932277 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:15.932186    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:15.932277 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:15.932217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:15.932396 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:15.932334    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:16.072839 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:16.072801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"79fe626121abd2fd8effd4ea12d13d6d0de8006e21a41bb274667b6e5a57f62e"}
Sep 09 03:06:16.932347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:16.932313    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:16.932567 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:16.932444    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:17.070827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:17.070792    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:06:17.071465 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:17.071441    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:06:17.932315 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:17.932285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:17.932650 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:17.932331    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:17.932650 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:17.932413    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:17.932650 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:17.932535    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:18.081116 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:18.081085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" event={"ID":"a66c6658-e660-435a-aec6-b3e64052cc35","Type":"ContainerStarted","Data":"7a3999ac4b15ba06624a2b93362e50de4cb0cefb265ec5323b8ed362acf4b715"}
Sep 09 03:06:18.081356 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:18.081334    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:06:18.099252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:18.099226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:06:18.104254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:18.104211    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6" podStartSLOduration=8.545396393 podStartE2EDuration="27.104198764s" totalImagesPullingTime="18.558802371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:18.103232044 +0000 UTC m=+26.665558589" watchObservedRunningTime="2026-09-09 03:06:18.104198764 +0000 UTC m=+26.666525309"
Sep 09 03:06:18.758488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:18.758281    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:06:18.931936 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:18.931900    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:18.932100 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:18.932005    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:19.085444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.085413    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2dbc178-5c81-4110-be6e-82b349c347fa" containerID="ac477b656dbd8ea7224a541f0b705a0c91e3002926883a59a74541a75e5b534c" exitCode=0
Sep 09 03:06:19.086133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.085509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" event={"ID":"f2dbc178-5c81-4110-be6e-82b349c347fa","Type":"ContainerDied","Data":"ac477b656dbd8ea7224a541f0b705a0c91e3002926883a59a74541a75e5b534c"}
Sep 09 03:06:19.086133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.086054    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:06:19.103180 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.103152    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:06:19.932036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.931991    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:19.932205 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.931991    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:19.932205 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:19.932119    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:19.932297 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:19.932205    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:19.990358 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.990319    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4drfs"]
Sep 09 03:06:19.993525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.993436    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w75g9"]
Sep 09 03:06:19.994120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.994099    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ccq7z"]
Sep 09 03:06:19.994227 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:19.994210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:19.994319 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:19.994295    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:20.087897 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:20.087866    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:20.088579 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:20.088004    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:20.088775 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:20.088750    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:20.088900 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:20.088877    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:21.092934 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:21.092833    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2dbc178-5c81-4110-be6e-82b349c347fa" containerID="4d6281445830e661d2a2f1b5e601ae2eb6105ceb27bbbc21258c69d509a36391" exitCode=0
Sep 09 03:06:21.092934 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:21.092921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" event={"ID":"f2dbc178-5c81-4110-be6e-82b349c347fa","Type":"ContainerDied","Data":"4d6281445830e661d2a2f1b5e601ae2eb6105ceb27bbbc21258c69d509a36391"}
Sep 09 03:06:21.590051 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:21.590016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:06:21.590330 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:21.590147    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 03:06:21.590568 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:21.590526    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q765z"
Sep 09 03:06:21.933755 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:21.933717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:21.933957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:21.933824    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:21.933957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:21.933869    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:21.933957 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:21.933888    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:21.933957 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:21.933943    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:21.934167 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:21.934036    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:23.100716 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:23.100679    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2dbc178-5c81-4110-be6e-82b349c347fa" containerID="e80ce908078f3d18dc62ed2392a1cbde9f1c6a64eb488b3bcd825a636feb7b95" exitCode=0
Sep 09 03:06:23.101079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:23.100715    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" event={"ID":"f2dbc178-5c81-4110-be6e-82b349c347fa","Type":"ContainerDied","Data":"e80ce908078f3d18dc62ed2392a1cbde9f1c6a64eb488b3bcd825a636feb7b95"}
Sep 09 03:06:23.932826 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:23.932731    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:23.932826 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:23.932781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:23.932826 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:23.932748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:23.933056 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:23.932850    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4drfs" podUID="8a2c4bfb-6825-4173-93a0-96a27cbdaedd"
Sep 09 03:06:23.933056 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:23.932974    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:06:23.933135 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:23.933063    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccq7z" podUID="2f0d4458-174a-415d-93cb-eb5fe526d927"
Sep 09 03:06:24.545673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:24.545628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:24.545700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:24.545745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:24.545775    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:24.545855    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret podName:8a2c4bfb-6825-4173-93a0-96a27cbdaedd nodeName:}" failed. No retries permitted until 2026-09-09 03:06:56.545832254 +0000 UTC m=+65.108158794 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret") pod "global-pull-secret-syncer-4drfs" (UID: "8a2c4bfb-6825-4173-93a0-96a27cbdaedd") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:24.545863    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:24.545891    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:24.545912    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:24.545917    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:56.545900593 +0000 UTC m=+65.108227134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:24.545924    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gs8ns for pod openshift-network-diagnostics/network-check-target-ccq7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:24.546201 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:24.545967    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns podName:2f0d4458-174a-415d-93cb-eb5fe526d927 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:56.545953249 +0000 UTC m=+65.108279772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs8ns" (UniqueName: "kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns") pod "network-check-target-ccq7z" (UID: "2f0d4458-174a-415d-93cb-eb5fe526d927") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:25.688199 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.688161    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeReady"
Sep 09 03:06:25.688715 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.688323    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 03:06:25.723516 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.723423    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56749b4d77-d7p5w"]
Sep 09 03:06:25.726026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.725998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.728048 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.728020    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 03:06:25.728219 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.728199    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nklpr\""
Sep 09 03:06:25.728290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.728228    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 03:06:25.728458 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.728445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 03:06:25.733462 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.733274    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 03:06:25.738105 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.738082    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56749b4d77-d7p5w"]
Sep 09 03:06:25.740777 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.740756    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6n2dq"]
Sep 09 03:06:25.743047 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.743027    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d466q"]
Sep 09 03:06:25.743165 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.743149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6n2dq"
Sep 09 03:06:25.744768 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.744748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.745970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.745945    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 03:06:25.745970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.745967    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 03:06:25.746155 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.746020    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fjjq4\""
Sep 09 03:06:25.746155 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.745945    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 03:06:25.746489 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.746470    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 03:06:25.746735 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.746715    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8s7tp\""
Sep 09 03:06:25.746823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.746777    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 03:06:25.752719 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.752694    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6n2dq"]
Sep 09 03:06:25.755096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.755071    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d466q"]
Sep 09 03:06:25.855187 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-certificates\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.855187 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-installation-pull-secrets\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.855413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb59c21-963b-4254-8d19-d2fb97b78882-tmp-dir\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.855413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-trusted-ca\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.855413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-bound-sa-token\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.855413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be59ed35-76ae-464d-b7ff-5c3ded0d76c5-cert\") pod \"ingress-canary-6n2dq\" (UID: \"be59ed35-76ae-464d-b7ff-5c3ded0d76c5\") " pod="openshift-ingress-canary/ingress-canary-6n2dq"
Sep 09 03:06:25.855413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbdrh\" (UniqueName: \"kubernetes.io/projected/bbb59c21-963b-4254-8d19-d2fb97b78882-kube-api-access-wbdrh\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.855413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5jkd\" (UniqueName: \"kubernetes.io/projected/be59ed35-76ae-464d-b7ff-5c3ded0d76c5-kube-api-access-v5jkd\") pod \"ingress-canary-6n2dq\" (UID: \"be59ed35-76ae-464d-b7ff-5c3ded0d76c5\") " pod="openshift-ingress-canary/ingress-canary-6n2dq"
Sep 09 03:06:25.855413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.855695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855427    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.855695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-image-registry-private-configuration\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.855695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79426f9f-2ce5-411d-85f5-3f74010b0ac6-ca-trust-extracted\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.855695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6xl\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-kube-api-access-kx6xl\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.855695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.855531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb59c21-963b-4254-8d19-d2fb97b78882-config-volume\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.932645 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.932607    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:25.932836 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.932608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:25.933030 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.932608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:25.935903 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.935874    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:06:25.936040 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.935902    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:06:25.936115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.936098    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bzb6q\""
Sep 09 03:06:25.936172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.936157    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:06:25.936227 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.936164    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qqmk2\""
Sep 09 03:06:25.936459 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.936441    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:06:25.956242 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-certificates\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.956388 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-installation-pull-secrets\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.956388 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb59c21-963b-4254-8d19-d2fb97b78882-tmp-dir\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.956493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-trusted-ca\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.956493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-bound-sa-token\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.956493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be59ed35-76ae-464d-b7ff-5c3ded0d76c5-cert\") pod \"ingress-canary-6n2dq\" (UID: \"be59ed35-76ae-464d-b7ff-5c3ded0d76c5\") " pod="openshift-ingress-canary/ingress-canary-6n2dq"
Sep 09 03:06:25.956615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbdrh\" (UniqueName: \"kubernetes.io/projected/bbb59c21-963b-4254-8d19-d2fb97b78882-kube-api-access-wbdrh\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.956615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956531    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5jkd\" (UniqueName: \"kubernetes.io/projected/be59ed35-76ae-464d-b7ff-5c3ded0d76c5-kube-api-access-v5jkd\") pod \"ingress-canary-6n2dq\" (UID: \"be59ed35-76ae-464d-b7ff-5c3ded0d76c5\") " pod="openshift-ingress-canary/ingress-canary-6n2dq"
Sep 09 03:06:25.956615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.956615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.956765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-image-registry-private-configuration\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.956765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79426f9f-2ce5-411d-85f5-3f74010b0ac6-ca-trust-extracted\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.956765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6xl\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-kube-api-access-kx6xl\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.956765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb59c21-963b-4254-8d19-d2fb97b78882-config-volume\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.956913 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb59c21-963b-4254-8d19-d2fb97b78882-tmp-dir\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.956913 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.956868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-certificates\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.957255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.957228    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb59c21-963b-4254-8d19-d2fb97b78882-config-volume\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.957558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.957513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79426f9f-2ce5-411d-85f5-3f74010b0ac6-ca-trust-extracted\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.958019 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:25.957997    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:25.958112 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:25.958094    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:06:25.958112 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:25.958110    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:06:25.958212 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:25.958111    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:26.458090738 +0000 UTC m=+35.020417277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:25.958212 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:25.958178    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:26.458160907 +0000 UTC m=+35.020487437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:06:25.958212 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.958015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-trusted-ca\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.961426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.961397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-installation-pull-secrets\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.961503 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.961403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be59ed35-76ae-464d-b7ff-5c3ded0d76c5-cert\") pod \"ingress-canary-6n2dq\" (UID: \"be59ed35-76ae-464d-b7ff-5c3ded0d76c5\") " pod="openshift-ingress-canary/ingress-canary-6n2dq"
Sep 09 03:06:25.961921 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.961899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-image-registry-private-configuration\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.967027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.967000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbdrh\" (UniqueName: \"kubernetes.io/projected/bbb59c21-963b-4254-8d19-d2fb97b78882-kube-api-access-wbdrh\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:25.967328 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.967306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5jkd\" (UniqueName: \"kubernetes.io/projected/be59ed35-76ae-464d-b7ff-5c3ded0d76c5-kube-api-access-v5jkd\") pod \"ingress-canary-6n2dq\" (UID: \"be59ed35-76ae-464d-b7ff-5c3ded0d76c5\") " pod="openshift-ingress-canary/ingress-canary-6n2dq"
Sep 09 03:06:25.967535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.967513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-bound-sa-token\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:25.967868 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:25.967823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6xl\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-kube-api-access-kx6xl\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:26.055939 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:26.055902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6n2dq"
Sep 09 03:06:26.459910 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:26.459814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:26.459910 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:26.459871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:26.460153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:26.459996    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:26.460153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:26.460061    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:27.460041821 +0000 UTC m=+36.022368366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:26.460153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:26.460057    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:06:26.460153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:26.460086    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:06:26.460327 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:26.460168    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:27.460148268 +0000 UTC m=+36.022474819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:06:27.321588 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:27.321370    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6n2dq"]
Sep 09 03:06:27.326404 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:06:27.326374    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe59ed35_76ae_464d_b7ff_5c3ded0d76c5.slice/crio-da309dbf9054ae8943416ad19ea2698a9cf107de7972cc4e2f7a29d371bf1343 WatchSource:0}: Error finding container da309dbf9054ae8943416ad19ea2698a9cf107de7972cc4e2f7a29d371bf1343: Status 404 returned error can't find the container with id da309dbf9054ae8943416ad19ea2698a9cf107de7972cc4e2f7a29d371bf1343
Sep 09 03:06:27.467821 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:27.467791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:27.467960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:27.467831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:27.467960 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:27.467946    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:06:27.467960 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:27.467958    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:06:27.468064 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:27.468011    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:29.467993168 +0000 UTC m=+38.030319712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:06:27.468108 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:27.468062    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:27.468157 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:27.468144    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:29.468124216 +0000 UTC m=+38.030450740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:28.116157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:28.116114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6n2dq" event={"ID":"be59ed35-76ae-464d-b7ff-5c3ded0d76c5","Type":"ContainerStarted","Data":"da309dbf9054ae8943416ad19ea2698a9cf107de7972cc4e2f7a29d371bf1343"}
Sep 09 03:06:28.119533 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:28.119502    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2dbc178-5c81-4110-be6e-82b349c347fa" containerID="ae35847f68130a98540d34c9670a95384852837785cf4c2710ed162eba4ac9d0" exitCode=0
Sep 09 03:06:28.119700 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:28.119584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" event={"ID":"f2dbc178-5c81-4110-be6e-82b349c347fa","Type":"ContainerDied","Data":"ae35847f68130a98540d34c9670a95384852837785cf4c2710ed162eba4ac9d0"}
Sep 09 03:06:29.125741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:29.125622    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2dbc178-5c81-4110-be6e-82b349c347fa" containerID="169004ece8460a12a34b469e43a375f977f698a4d3f2e323b3e818d4a3d20d3b" exitCode=0
Sep 09 03:06:29.125741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:29.125662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" event={"ID":"f2dbc178-5c81-4110-be6e-82b349c347fa","Type":"ContainerDied","Data":"169004ece8460a12a34b469e43a375f977f698a4d3f2e323b3e818d4a3d20d3b"}
Sep 09 03:06:29.485914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:29.485865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:29.485914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:29.485917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:29.486153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:29.486051    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:06:29.486153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:29.486061    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:29.486153 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:29.486151    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:33.486131639 +0000 UTC m=+42.048458184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:29.486297 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:29.486067    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:06:29.486297 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:29.486260    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:33.486241283 +0000 UTC m=+42.048567809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:06:30.131258 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:30.131220    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" event={"ID":"f2dbc178-5c81-4110-be6e-82b349c347fa","Type":"ContainerStarted","Data":"4befba3bec8519ccd5f6ef009b8687bc937d4b0d1489b36dc79575b34b48dcfa"}
Sep 09 03:06:30.153874 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:30.153818    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bgn7l" podStartSLOduration=8.681859962 podStartE2EDuration="38.153802319s" totalImagesPullingTime="29.471942357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:30.151798106 +0000 UTC m=+38.714124650" watchObservedRunningTime="2026-09-09 03:06:30.153802319 +0000 UTC m=+38.716128855"
Sep 09 03:06:31.045915 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:31.045867    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:31.129588 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:31.129522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:31.134563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:31.134519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6n2dq" event={"ID":"be59ed35-76ae-464d-b7ff-5c3ded0d76c5","Type":"ContainerStarted","Data":"1f98e1fb3b03dc4183bd38ad35163969ae5839f8111f0bd24f44b305eb9a3e7e"}
Sep 09 03:06:31.148557 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:31.148354    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6n2dq" podStartSLOduration=3.071526851 podStartE2EDuration="6.14833597s" totalImagesPullingTime="3.076809119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:31.147374631 +0000 UTC m=+39.709701188" watchObservedRunningTime="2026-09-09 03:06:31.14833597 +0000 UTC m=+39.710662516"
Sep 09 03:06:31.779075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:31.779035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:31.879125 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:31.879096    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:32.206221 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:32.206188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:32.257484 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:32.257452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:33.517668 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:33.517626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:33.517668 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:33.517667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:33.518083 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:33.517788    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:06:33.518083 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:33.517788    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:33.518083 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:33.517860    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:41.517842738 +0000 UTC m=+50.080169289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:33.518083 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:33.517799    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:06:33.518083 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:33.517939    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:41.517927319 +0000 UTC m=+50.080253841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:06:36.412729 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:36.412695    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:36.512583 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:36.512550    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:37.037253 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:37.037215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:37.092104 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:37.092055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:41.575048 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:41.575011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:41.575048 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:41.575050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:41.575455 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:41.575162    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:41.575455 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:41.575233    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:57.575216114 +0000 UTC m=+66.137542636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:41.575455 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:41.575164    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:06:41.575455 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:41.575271    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:06:41.575455 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:41.575331    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:57.575316705 +0000 UTC m=+66.137643256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:06:51.105250 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:51.105217    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlm6"
Sep 09 03:06:51.256919 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:51.256885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:51.305436 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:51.305403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:52.370501 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:52.370467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:52.430943 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:52.430912    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:56.586875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.586826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:56.586875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.586879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:56.587304 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.587019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:06:56.589081 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.589058    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:06:56.589260 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.589239    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:06:56.589310 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.589293    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:06:56.598009 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:56.597985    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:06:56.598142 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:56.598058    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:08:00.598040804 +0000 UTC m=+129.160367326 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : secret "metrics-daemon-secret" not found
Sep 09 03:06:56.599734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.599700    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:06:56.601240 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.601212    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a2c4bfb-6825-4173-93a0-96a27cbdaedd-original-pull-secret\") pod \"global-pull-secret-syncer-4drfs\" (UID: \"8a2c4bfb-6825-4173-93a0-96a27cbdaedd\") " pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:56.610709 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.610665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8ns\" (UniqueName: \"kubernetes.io/projected/2f0d4458-174a-415d-93cb-eb5fe526d927-kube-api-access-gs8ns\") pod \"network-check-target-ccq7z\" (UID: \"2f0d4458-174a-415d-93cb-eb5fe526d927\") " pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:56.848991 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.848901    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4drfs"
Sep 09 03:06:56.865175 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.865149    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qqmk2\""
Sep 09 03:06:56.873817 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:56.873787    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:06:57.120010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.119923    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4drfs"]
Sep 09 03:06:57.126362 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:06:57.126336    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2c4bfb_6825_4173_93a0_96a27cbdaedd.slice/crio-7d4a3b9cdfb7c55350c538c47ca9049fbe463620df2ab96fa2b66bdafbd45ec5 WatchSource:0}: Error finding container 7d4a3b9cdfb7c55350c538c47ca9049fbe463620df2ab96fa2b66bdafbd45ec5: Status 404 returned error can't find the container with id 7d4a3b9cdfb7c55350c538c47ca9049fbe463620df2ab96fa2b66bdafbd45ec5
Sep 09 03:06:57.152259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.152230    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ccq7z"]
Sep 09 03:06:57.155510 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:06:57.155475    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0d4458_174a_415d_93cb_eb5fe526d927.slice/crio-bb4222bffb0d2f7150b6fc61d1c5cca3bb95c44e7bdcc1c8c6167b1902a04667 WatchSource:0}: Error finding container bb4222bffb0d2f7150b6fc61d1c5cca3bb95c44e7bdcc1c8c6167b1902a04667: Status 404 returned error can't find the container with id bb4222bffb0d2f7150b6fc61d1c5cca3bb95c44e7bdcc1c8c6167b1902a04667
Sep 09 03:06:57.197549 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.197510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ccq7z" event={"ID":"2f0d4458-174a-415d-93cb-eb5fe526d927","Type":"ContainerStarted","Data":"bb4222bffb0d2f7150b6fc61d1c5cca3bb95c44e7bdcc1c8c6167b1902a04667"}
Sep 09 03:06:57.198731 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.198709    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4drfs" event={"ID":"8a2c4bfb-6825-4173-93a0-96a27cbdaedd","Type":"ContainerStarted","Data":"7d4a3b9cdfb7c55350c538c47ca9049fbe463620df2ab96fa2b66bdafbd45ec5"}
Sep 09 03:06:57.342660 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.342625    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:57.400244 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.400166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:57.456254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.456219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:06:57.595290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.595252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:06:57.595290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:06:57.595293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:06:57.595714 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:57.595400    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:57.595714 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:57.595409    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:06:57.595714 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:57.595424    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:06:57.595714 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:57.595465    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:07:29.595449177 +0000 UTC m=+98.157775700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:57.595714 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:06:57.595481    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:07:29.595473776 +0000 UTC m=+98.157800299 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:07:03.217061 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:03.217024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4drfs" event={"ID":"8a2c4bfb-6825-4173-93a0-96a27cbdaedd","Type":"ContainerStarted","Data":"97ed50e0c4325d97d8f35a1836a707c8d155745621e4521eac2286bcb5075848"}
Sep 09 03:07:03.218441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:03.218414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ccq7z" event={"ID":"2f0d4458-174a-415d-93cb-eb5fe526d927","Type":"ContainerStarted","Data":"e104755089659c8a7cc756cbf39884acf656ecde6a6be3ac9285f38cc7cbab19"}
Sep 09 03:07:03.218577 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:03.218555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:07:03.230800 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:03.230746    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4drfs" podStartSLOduration=65.863872371 podStartE2EDuration="1m11.23072777s" totalImagesPullingTime="5.366855399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:03.230012021 +0000 UTC m=+71.792338568" watchObservedRunningTime="2026-09-09 03:07:03.23072777 +0000 UTC m=+71.793054316"
Sep 09 03:07:03.242161 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:03.242120    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ccq7z" podStartSLOduration=65.908128676 podStartE2EDuration="1m11.242107045s" totalImagesPullingTime="5.333978369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:03.241692334 +0000 UTC m=+71.804018889" watchObservedRunningTime="2026-09-09 03:07:03.242107045 +0000 UTC m=+71.804433589"
Sep 09 03:07:05.898868 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:05.898835    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:07:05.950723 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:05.950690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:07:29.623373 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:29.623314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:07:29.623373 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:29.623375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:07:29.623897 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:07:29.623473    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:07:29.623897 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:07:29.623485    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:07:29.623897 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:07:29.623490    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:07:29.623897 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:07:29.623559    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:08:33.623525063 +0000 UTC m=+162.185851586 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:07:29.623897 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:07:29.623582    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:08:33.623565153 +0000 UTC m=+162.185891677 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:07:34.223238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:07:34.223200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ccq7z"
Sep 09 03:08:00.644418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:08:00.644365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:08:00.644951 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:00.644517    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:08:00.644951 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:00.644634    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs podName:c99b1fee-9619-4012-8bf1-19657cc1ac91 nodeName:}" failed. No retries permitted until 2026-09-09 03:10:02.644608762 +0000 UTC m=+251.206935286 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs") pod "network-metrics-daemon-w75g9" (UID: "c99b1fee-9619-4012-8bf1-19657cc1ac91") : secret "metrics-daemon-secret" not found
Sep 09 03:08:28.737698 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:28.737645    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" podUID="79426f9f-2ce5-411d-85f5-3f74010b0ac6"
Sep 09 03:08:28.762137 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:28.762085    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d466q" podUID="bbb59c21-963b-4254-8d19-d2fb97b78882"
Sep 09 03:08:28.955582 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:28.955519    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-w75g9" podUID="c99b1fee-9619-4012-8bf1-19657cc1ac91"
Sep 09 03:08:29.441002 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:08:29.440969    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:08:29.441002 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:08:29.441011    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d466q"
Sep 09 03:08:33.684164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:08:33.684105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:08:33.684164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:08:33.684159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:08:33.684727 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:33.684269    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:08:33.684727 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:33.684266    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:08:33.684727 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:33.684350    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls podName:bbb59c21-963b-4254-8d19-d2fb97b78882 nodeName:}" failed. No retries permitted until 2026-09-09 03:10:35.68433487 +0000 UTC m=+284.246661411 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls") pod "dns-default-d466q" (UID: "bbb59c21-963b-4254-8d19-d2fb97b78882") : secret "dns-default-metrics-tls" not found
Sep 09 03:08:33.684727 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:33.684280    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56749b4d77-d7p5w: secret "image-registry-tls" not found
Sep 09 03:08:33.684727 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:08:33.684422    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls podName:79426f9f-2ce5-411d-85f5-3f74010b0ac6 nodeName:}" failed. No retries permitted until 2026-09-09 03:10:35.684408897 +0000 UTC m=+284.246735421 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls") pod "image-registry-56749b4d77-d7p5w" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6") : secret "image-registry-tls" not found
Sep 09 03:08:37.216150 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:08:37.216113    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:08:40.932040 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:08:40.931987    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:09:10.263872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:10.263788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:36.050230 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.050197    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4t4zd"]
Sep 09 03:09:36.051986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.051969    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:36.055876 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.055857    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 03:09:36.055876 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.055868    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d8fbz\""
Sep 09 03:09:36.057264 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.057244    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 03:09:36.076586 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.076533    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4t4zd"]
Sep 09 03:09:36.077165 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.077146    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"]
Sep 09 03:09:36.078882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.078849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.080964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.080941    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.081077 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.080976    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.081569 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.081531    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 03:09:36.081666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.081573    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 03:09:36.088571 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.088529    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"]
Sep 09 03:09:36.130659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.130626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:36.130659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.130662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4366ede7-855d-48f2-8285-e1fcf0e925ec-nginx-conf\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:36.143936 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.143906    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-794f5b986b-77gpk"]
Sep 09 03:09:36.145622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.145596    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.148039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.148022    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 03:09:36.148158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.148127    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.148495 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.148473    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 03:09:36.148620 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.148522    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 03:09:36.148620 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.148553    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sc5sl\""
Sep 09 03:09:36.148620 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.148531    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.148620 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.148594    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 03:09:36.161847 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.161812    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-794f5b986b-77gpk"]
Sep 09 03:09:36.231893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.231850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/753a60ca-4432-4758-8739-f0c3e0be1433-tmp\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.231893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.231890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-stats-auth\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.231915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.232008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-default-certificate\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.232048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.232074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/753a60ca-4432-4758-8739-f0c3e0be1433-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.232101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.232123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4366ede7-855d-48f2-8285-e1fcf0e925ec-nginx-conf\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.232151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxn9m\" (UniqueName: \"kubernetes.io/projected/753a60ca-4432-4758-8739-f0c3e0be1433-kube-api-access-cxn9m\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.232167    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:09:36.232172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.232173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snmbb\" (UniqueName: \"kubernetes.io/projected/7042025b-b615-4b6e-a06c-eda170b933bb-kube-api-access-snmbb\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.232564 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.232318    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert podName:4366ede7-855d-48f2-8285-e1fcf0e925ec nodeName:}" failed. No retries permitted until 2026-09-09 03:09:36.732295801 +0000 UTC m=+225.294622342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4t4zd" (UID: "4366ede7-855d-48f2-8285-e1fcf0e925ec") : secret "networking-console-plugin-cert" not found
Sep 09 03:09:36.232875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.232855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4366ede7-855d-48f2-8285-e1fcf0e925ec-nginx-conf\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:36.241889 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.241859    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"]
Sep 09 03:09:36.243618 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.243602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.248575 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.246191    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-hrtwd"]
Sep 09 03:09:36.248575 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.246215    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 03:09:36.248575 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.246313    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.248575 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.246325    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xc5hh\""
Sep 09 03:09:36.248575 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.246630    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.248575 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.247881    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 03:09:36.250353 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.250241    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-hrtwd"
Sep 09 03:09:36.252490 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.252470    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xsczf\""
Sep 09 03:09:36.253992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.253962    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"]
Sep 09 03:09:36.256862 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.256813    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-hrtwd"]
Sep 09 03:09:36.331364 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.331269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:36.333168 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333146    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/753a60ca-4432-4758-8739-f0c3e0be1433-tmp\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.333220 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-stats-auth\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.333220 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.333295 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.333283    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 03:09:36.333348 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c34dac72-7fef-47aa-889f-67f866f4f1e6-config\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.333382 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.333355    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:36.833336817 +0000 UTC m=+225.395663343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : secret "router-metrics-certs-default" not found
Sep 09 03:09:36.333422 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-default-certificate\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.333475 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/753a60ca-4432-4758-8739-f0c3e0be1433-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.333559 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.333559 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxn9m\" (UniqueName: \"kubernetes.io/projected/753a60ca-4432-4758-8739-f0c3e0be1433-kube-api-access-cxn9m\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.333559 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/753a60ca-4432-4758-8739-f0c3e0be1433-tmp\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.333740 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.333662    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:36.833642615 +0000 UTC m=+225.395969156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : configmap references non-existent config key: service-ca.crt
Sep 09 03:09:36.333740 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snmbb\" (UniqueName: \"kubernetes.io/projected/7042025b-b615-4b6e-a06c-eda170b933bb-kube-api-access-snmbb\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.333740 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzrss\" (UniqueName: \"kubernetes.io/projected/33eb3367-494f-456c-a6fe-139aa66e2f2e-kube-api-access-dzrss\") pod \"network-check-source-685456655f-hrtwd\" (UID: \"33eb3367-494f-456c-a6fe-139aa66e2f2e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-hrtwd"
Sep 09 03:09:36.333920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c34dac72-7fef-47aa-889f-67f866f4f1e6-serving-cert\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.333920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.333800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvcjg\" (UniqueName: \"kubernetes.io/projected/c34dac72-7fef-47aa-889f-67f866f4f1e6-kube-api-access-dvcjg\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.336234 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.336203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/753a60ca-4432-4758-8739-f0c3e0be1433-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.336352 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.336337    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-stats-auth\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.336459 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.336440    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-default-certificate\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.343362 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.343332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxn9m\" (UniqueName: \"kubernetes.io/projected/753a60ca-4432-4758-8739-f0c3e0be1433-kube-api-access-cxn9m\") pod \"klusterlet-addon-workmgr-5b8546cd57-6477t\" (UID: \"753a60ca-4432-4758-8739-f0c3e0be1433\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.343795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.343773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snmbb\" (UniqueName: \"kubernetes.io/projected/7042025b-b615-4b6e-a06c-eda170b933bb-kube-api-access-snmbb\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.388522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.388483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:36.434968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.434678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c34dac72-7fef-47aa-889f-67f866f4f1e6-config\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.434968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.434797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzrss\" (UniqueName: \"kubernetes.io/projected/33eb3367-494f-456c-a6fe-139aa66e2f2e-kube-api-access-dzrss\") pod \"network-check-source-685456655f-hrtwd\" (UID: \"33eb3367-494f-456c-a6fe-139aa66e2f2e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-hrtwd"
Sep 09 03:09:36.434968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.434838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c34dac72-7fef-47aa-889f-67f866f4f1e6-serving-cert\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.434968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.434860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvcjg\" (UniqueName: \"kubernetes.io/projected/c34dac72-7fef-47aa-889f-67f866f4f1e6-kube-api-access-dvcjg\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.435893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.435863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c34dac72-7fef-47aa-889f-67f866f4f1e6-config\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.438503 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.438476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c34dac72-7fef-47aa-889f-67f866f4f1e6-serving-cert\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.442678 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.442656    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvcjg\" (UniqueName: \"kubernetes.io/projected/c34dac72-7fef-47aa-889f-67f866f4f1e6-kube-api-access-dvcjg\") pod \"service-ca-operator-859f94c4b-j5422\" (UID: \"c34dac72-7fef-47aa-889f-67f866f4f1e6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.442821 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.442800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzrss\" (UniqueName: \"kubernetes.io/projected/33eb3367-494f-456c-a6fe-139aa66e2f2e-kube-api-access-dzrss\") pod \"network-check-source-685456655f-hrtwd\" (UID: \"33eb3367-494f-456c-a6fe-139aa66e2f2e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-hrtwd"
Sep 09 03:09:36.469115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.469083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:36.555991 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.555953    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"
Sep 09 03:09:36.562343 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.562322    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-hrtwd"
Sep 09 03:09:36.655454 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.655420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"]
Sep 09 03:09:36.738443 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.738401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:36.738621 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.738581    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:09:36.738668 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.738650    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert podName:4366ede7-855d-48f2-8285-e1fcf0e925ec nodeName:}" failed. No retries permitted until 2026-09-09 03:09:37.738635392 +0000 UTC m=+226.300961916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4t4zd" (UID: "4366ede7-855d-48f2-8285-e1fcf0e925ec") : secret "networking-console-plugin-cert" not found
Sep 09 03:09:36.823411 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.823376    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422"]
Sep 09 03:09:36.826479 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:09:36.826434    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34dac72_7fef_47aa_889f_67f866f4f1e6.slice/crio-ade13458b564c0bc4763e92e46393836753cdd8508d239453a3076f3d0bef5a4 WatchSource:0}: Error finding container ade13458b564c0bc4763e92e46393836753cdd8508d239453a3076f3d0bef5a4: Status 404 returned error can't find the container with id ade13458b564c0bc4763e92e46393836753cdd8508d239453a3076f3d0bef5a4
Sep 09 03:09:36.826798 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.826776    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-hrtwd"]
Sep 09 03:09:36.829974 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:09:36.829954    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33eb3367_494f_456c_a6fe_139aa66e2f2e.slice/crio-98c11a6cf4ec52824f253f4d5a422a7d675b60067bf37fa0134b60ceebea8f19 WatchSource:0}: Error finding container 98c11a6cf4ec52824f253f4d5a422a7d675b60067bf37fa0134b60ceebea8f19: Status 404 returned error can't find the container with id 98c11a6cf4ec52824f253f4d5a422a7d675b60067bf37fa0134b60ceebea8f19
Sep 09 03:09:36.839409 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.839386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.839492 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:36.839443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:36.839580 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.839527    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 03:09:36.839638 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.839593    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:37.839574565 +0000 UTC m=+226.401901109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : configmap references non-existent config key: service-ca.crt
Sep 09 03:09:36.839638 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:36.839608    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:37.839601744 +0000 UTC m=+226.401928267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : secret "router-metrics-certs-default" not found
Sep 09 03:09:37.609786 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:37.609744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-hrtwd" event={"ID":"33eb3367-494f-456c-a6fe-139aa66e2f2e","Type":"ContainerStarted","Data":"e86f041881835cbf5f2e7073354216d35edd422ef9c536d428fb3073bc27ee23"}
Sep 09 03:09:37.610263 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:37.609817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-hrtwd" event={"ID":"33eb3367-494f-456c-a6fe-139aa66e2f2e","Type":"ContainerStarted","Data":"98c11a6cf4ec52824f253f4d5a422a7d675b60067bf37fa0134b60ceebea8f19"}
Sep 09 03:09:37.612248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:37.612195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t" event={"ID":"753a60ca-4432-4758-8739-f0c3e0be1433","Type":"ContainerStarted","Data":"977343597a2d3e46da700b66304e84c8a01dd12ee70937950e089f5ae9662a25"}
Sep 09 03:09:37.614725 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:37.614672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422" event={"ID":"c34dac72-7fef-47aa-889f-67f866f4f1e6","Type":"ContainerStarted","Data":"ade13458b564c0bc4763e92e46393836753cdd8508d239453a3076f3d0bef5a4"}
Sep 09 03:09:37.625453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:37.624396    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-hrtwd" podStartSLOduration=1.6243737459999998 podStartE2EDuration="1.624373746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:37.623754946 +0000 UTC m=+226.186081494" watchObservedRunningTime="2026-09-09 03:09:37.624373746 +0000 UTC m=+226.186700305"
Sep 09 03:09:37.746800 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:37.746747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:37.746994 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:37.746916    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:09:37.747055 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:37.746999    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert podName:4366ede7-855d-48f2-8285-e1fcf0e925ec nodeName:}" failed. No retries permitted until 2026-09-09 03:09:39.746975306 +0000 UTC m=+228.309301852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4t4zd" (UID: "4366ede7-855d-48f2-8285-e1fcf0e925ec") : secret "networking-console-plugin-cert" not found
Sep 09 03:09:37.847430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:37.847391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:37.847641 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:37.847503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:37.847641 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:37.847510    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 03:09:37.847641 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:37.847605    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:39.847585588 +0000 UTC m=+228.409912136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : secret "router-metrics-certs-default" not found
Sep 09 03:09:37.847815 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:37.847655    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:39.847635157 +0000 UTC m=+228.409961690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : configmap references non-existent config key: service-ca.crt
Sep 09 03:09:38.813610 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:38.813516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:39.622199 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:39.622106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422" event={"ID":"c34dac72-7fef-47aa-889f-67f866f4f1e6","Type":"ContainerStarted","Data":"9de227aefef083e15a3c31b591c29e510eec1d437ab346ab47206b29a1c7f37b"}
Sep 09 03:09:39.637332 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:39.637277    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422" podStartSLOduration=1.163133415 podStartE2EDuration="3.637252246s" totalImagesPullingTime="2.474118831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:39.636689447 +0000 UTC m=+228.199015993" watchObservedRunningTime="2026-09-09 03:09:39.637252246 +0000 UTC m=+228.199578792"
Sep 09 03:09:39.762393 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:39.762348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:39.762617 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:39.762468    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:09:39.762617 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:39.762532    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert podName:4366ede7-855d-48f2-8285-e1fcf0e925ec nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.762513647 +0000 UTC m=+232.324840193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4t4zd" (UID: "4366ede7-855d-48f2-8285-e1fcf0e925ec") : secret "networking-console-plugin-cert" not found
Sep 09 03:09:39.863587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:39.863551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:39.863979 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:39.863642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:39.863979 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:39.863698    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 03:09:39.863979 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:39.863763    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.863745875 +0000 UTC m=+232.426072500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : configmap references non-existent config key: service-ca.crt
Sep 09 03:09:39.863979 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:39.863779    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.863772614 +0000 UTC m=+232.426099137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : secret "router-metrics-certs-default" not found
Sep 09 03:09:40.529986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:40.529935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:40.580174 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:40.580137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:40.983259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:40.983193    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:41.084911 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:41.084867    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:41.236258 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:41.236171    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:41.340164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:41.340125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:41.631168 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:41.631133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t" event={"ID":"753a60ca-4432-4758-8739-f0c3e0be1433","Type":"ContainerStarted","Data":"62e13b26e895bec01fcb680fe5da08b3293d0d74e219553b65cfc3b1315ec2cb"}
Sep 09 03:09:41.631345 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:41.631265    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:41.632824 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:41.632805    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t"
Sep 09 03:09:41.646222 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:41.646175    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b8546cd57-6477t" podStartSLOduration=0.786873975 podStartE2EDuration="5.646156621s" totalImagesPullingTime="4.859282646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:41.644343509 +0000 UTC m=+230.206670053" watchObservedRunningTime="2026-09-09 03:09:41.646156621 +0000 UTC m=+230.208483158"
Sep 09 03:09:42.212945 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.212908    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2ggsw"]
Sep 09 03:09:42.215984 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.215958    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.217938 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.217904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:42.218094 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.217921    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 03:09:42.218094 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.217977    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 03:09:42.218094 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.218042    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 03:09:42.218237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.218131    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 03:09:42.218818 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.218782    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89spd\""
Sep 09 03:09:42.224883 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.224857    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ggsw"]
Sep 09 03:09:42.284677 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.284639    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-data-volume\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.284855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.284687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.284855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.284814    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-crio-socket\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.284949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.284853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6sx9\" (UniqueName: \"kubernetes.io/projected/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-api-access-z6sx9\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.284949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.284934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.385929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.385885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.386120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.385986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-crio-socket\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.386120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.386027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6sx9\" (UniqueName: \"kubernetes.io/projected/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-api-access-z6sx9\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.386120 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:42.386050    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.386120 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:42.386116    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:42.886095592 +0000 UTC m=+231.448422137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.386327 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.386121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-crio-socket\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.386327 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.386148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.386327 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.386189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-data-volume\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.386327 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:42.386200    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.386327 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:42.386247    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:42.886226762 +0000 UTC m=+231.448553286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.386601 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.386504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-data-volume\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.396635 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.396604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6sx9\" (UniqueName: \"kubernetes.io/projected/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-api-access-z6sx9\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.758783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.758744    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:42.853135 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.853097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:42.870017 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.869983    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dcdn4"]
Sep 09 03:09:42.873009 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.872992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:42.875132 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.875104    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 03:09:42.875257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.875112    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 03:09:42.875257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.875147    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 03:09:42.875257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.875166    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 03:09:42.875787 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.875767    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-z7mvn\""
Sep 09 03:09:42.881655 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.881634    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dcdn4"]
Sep 09 03:09:42.890047 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.890018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.890164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.890129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:42.890214 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:42.890164    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.890214 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:42.890192    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.890294 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:42.890233    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.890217316 +0000 UTC m=+232.452543840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.890294 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:42.890248    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.890241209 +0000 UTC m=+232.452567731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.953685 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.953645    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:42.991389 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.991352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4c36b4a-191a-45b7-92be-aae630307cac-signing-key\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:42.991590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.991535    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kdhk\" (UniqueName: \"kubernetes.io/projected/c4c36b4a-191a-45b7-92be-aae630307cac-kube-api-access-6kdhk\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:42.991654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:42.991592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4c36b4a-191a-45b7-92be-aae630307cac-signing-cabundle\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:43.092795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.092756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kdhk\" (UniqueName: \"kubernetes.io/projected/c4c36b4a-191a-45b7-92be-aae630307cac-kube-api-access-6kdhk\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:43.092964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.092807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4c36b4a-191a-45b7-92be-aae630307cac-signing-cabundle\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:43.093094 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.093069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4c36b4a-191a-45b7-92be-aae630307cac-signing-key\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:43.093488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.093468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4c36b4a-191a-45b7-92be-aae630307cac-signing-cabundle\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:43.095621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.095588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4c36b4a-191a-45b7-92be-aae630307cac-signing-key\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:43.101941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.101919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kdhk\" (UniqueName: \"kubernetes.io/projected/c4c36b4a-191a-45b7-92be-aae630307cac-kube-api-access-6kdhk\") pod \"service-ca-75d99c484-dcdn4\" (UID: \"c4c36b4a-191a-45b7-92be-aae630307cac\") " pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:43.183033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.182992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-dcdn4"
Sep 09 03:09:43.454457 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.454371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dcdn4"]
Sep 09 03:09:43.456983 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:09:43.456954    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c36b4a_191a_45b7_92be_aae630307cac.slice/crio-2618e9ccc02a8dc46209651d22f337a2863afc9937c4c88a617b02227ba7a017 WatchSource:0}: Error finding container 2618e9ccc02a8dc46209651d22f337a2863afc9937c4c88a617b02227ba7a017: Status 404 returned error can't find the container with id 2618e9ccc02a8dc46209651d22f337a2863afc9937c4c88a617b02227ba7a017
Sep 09 03:09:43.638726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.638686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-dcdn4" event={"ID":"c4c36b4a-191a-45b7-92be-aae630307cac","Type":"ContainerStarted","Data":"cc0784c6568f4759369528423d2ba027173f4a077e1d913d63ecbfe9af3b3e32"}
Sep 09 03:09:43.638897 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.638741    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-dcdn4" event={"ID":"c4c36b4a-191a-45b7-92be-aae630307cac","Type":"ContainerStarted","Data":"2618e9ccc02a8dc46209651d22f337a2863afc9937c4c88a617b02227ba7a017"}
Sep 09 03:09:43.655947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.655891    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-dcdn4" podStartSLOduration=1.655869768 podStartE2EDuration="1.655869768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:43.654593782 +0000 UTC m=+232.216920325" watchObservedRunningTime="2026-09-09 03:09:43.655869768 +0000 UTC m=+232.218196314"
Sep 09 03:09:43.799827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.799785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:43.800033 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.799899    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:09:43.800033 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.799972    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert podName:4366ede7-855d-48f2-8285-e1fcf0e925ec nodeName:}" failed. No retries permitted until 2026-09-09 03:09:51.799952176 +0000 UTC m=+240.362278739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4t4zd" (UID: "4366ede7-855d-48f2-8285-e1fcf0e925ec") : secret "networking-console-plugin-cert" not found
Sep 09 03:09:43.806561 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.806523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:43.901034 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.900993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:43.901235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.901048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:43.901235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.901116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:43.901235 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.901145    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 03:09:43.901235 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.901167    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:51.901146233 +0000 UTC m=+240.463472788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : configmap references non-existent config key: service-ca.crt
Sep 09 03:09:43.901235 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.901197    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs podName:7042025b-b615-4b6e-a06c-eda170b933bb nodeName:}" failed. No retries permitted until 2026-09-09 03:09:51.90118487 +0000 UTC m=+240.463511394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs") pod "router-default-794f5b986b-77gpk" (UID: "7042025b-b615-4b6e-a06c-eda170b933bb") : secret "router-metrics-certs-default" not found
Sep 09 03:09:43.901235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:43.901151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:43.901235 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.901215    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:43.901235 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.901225    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:43.901733 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.901269    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:45.901258179 +0000 UTC m=+234.463584720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:43.901733 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:43.901284    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:45.901276899 +0000 UTC m=+234.463603421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:44.174374 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:44.174262    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lplwr_226d6682-de69-414a-bec1-1a3114c89844/dns-node-resolver/0.log"
Sep 09 03:09:44.700153 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:44.700122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:44.863338 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:44.863301    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:44.987497 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:44.987413    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:45.088906 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:45.088866    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:45.189886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:45.189854    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:45.374321 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:45.374295    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwtvs_781959c0-527d-4192-86a4-99be89ec07ab/node-ca/0.log"
Sep 09 03:09:45.395068 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:45.395030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:45.774235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:45.774155    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6n2dq_be59ed35-76ae-464d-b7ff-5c3ded0d76c5/serve-healthcheck-canary/0.log"
Sep 09 03:09:45.917448 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:45.917400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:45.917448 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:45.917455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:45.917684 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:45.917527    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:45.917684 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:45.917623    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:49.917606834 +0000 UTC m=+238.479933357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:45.917684 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:45.917636    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:45.917798 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:45.917688    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:49.917671812 +0000 UTC m=+238.479998344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:46.394612 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:46.394508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:46.601314 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:46.601278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:46.699485 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:46.699382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:47.001098 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:47.001007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:47.205866 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:47.205834    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:47.793339 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:47.793306    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:48.396677 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:48.396638    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:48.497275 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:48.497237    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:48.601501 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:48.601464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:48.698145 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:48.698061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:49.000732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:49.000638    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:49.198904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:49.198865    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:49.795949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:49.795908    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:49.953520 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:49.953480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:49.953520 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:49.953522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:49.953788 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:49.953616    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:49.953788 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:49.953640    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:49.953788 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:49.953682    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:57.953670126 +0000 UTC m=+246.515996648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:49.953788 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:49.953695    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm podName:d0855e8e-3df7-41a5-8d2d-eaad6ff17404 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:57.953689157 +0000 UTC m=+246.516015680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ggsw" (UID: "d0855e8e-3df7-41a5-8d2d-eaad6ff17404") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:49.994135 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:49.994098    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:50.094967 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:50.094870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:50.594598 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:50.594559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:51.111797 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.111763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:51.213747 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.213717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:51.400630 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.400515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:51.797882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.797845    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:51.869974 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.869940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:09:51.870099 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:51.870084    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:09:51.870154 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:09:51.870140    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert podName:4366ede7-855d-48f2-8285-e1fcf0e925ec nodeName:}" failed. No retries permitted until 2026-09-09 03:10:07.870124693 +0000 UTC m=+256.432451216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4t4zd" (UID: "4366ede7-855d-48f2-8285-e1fcf0e925ec") : secret "networking-console-plugin-cert" not found
Sep 09 03:09:51.970641 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.970606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:51.970831 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.970658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:51.971479 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.971391    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7042025b-b615-4b6e-a06c-eda170b933bb-service-ca-bundle\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:51.973320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:51.973292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7042025b-b615-4b6e-a06c-eda170b933bb-metrics-certs\") pod \"router-default-794f5b986b-77gpk\" (UID: \"7042025b-b615-4b6e-a06c-eda170b933bb\") " pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:52.000391 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:52.000356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:52.056656 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:52.056566    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sc5sl\""
Sep 09 03:09:52.065384 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:52.065354    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:52.348304 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:52.348229    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-794f5b986b-77gpk"]
Sep 09 03:09:52.351673 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:09:52.351647    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7042025b_b615_4b6e_a06c_eda170b933bb.slice/crio-8ad54a15927ccf0932177314ac05cc9a4a02a393c8c5e90f4098908da4a57233 WatchSource:0}: Error finding container 8ad54a15927ccf0932177314ac05cc9a4a02a393c8c5e90f4098908da4a57233: Status 404 returned error can't find the container with id 8ad54a15927ccf0932177314ac05cc9a4a02a393c8c5e90f4098908da4a57233
Sep 09 03:09:52.593735 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:52.593706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:52.668845 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:52.668759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-794f5b986b-77gpk" event={"ID":"7042025b-b615-4b6e-a06c-eda170b933bb","Type":"ContainerStarted","Data":"33531649b409b5ed58f11527fdd89633a1cc777ddbca11cf507cf21841b456cb"}
Sep 09 03:09:52.668845 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:52.668819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-794f5b986b-77gpk" event={"ID":"7042025b-b615-4b6e-a06c-eda170b933bb","Type":"ContainerStarted","Data":"8ad54a15927ccf0932177314ac05cc9a4a02a393c8c5e90f4098908da4a57233"}
Sep 09 03:09:52.685209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:52.685156    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-794f5b986b-77gpk" podStartSLOduration=16.685136779 podStartE2EDuration="16.685136779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:52.684201578 +0000 UTC m=+241.246528124" watchObservedRunningTime="2026-09-09 03:09:52.685136779 +0000 UTC m=+241.247463336"
Sep 09 03:09:53.065811 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:53.065758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:53.068289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:53.068266    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:53.193928 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:53.193889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:53.389855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:53.389744    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:53.596529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:53.596486    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:53.672126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:53.672034    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:53.674671 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:53.674644    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-794f5b986b-77gpk"
Sep 09 03:09:55.239560 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:55.239504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:55.391039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:55.391008    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:55.796103 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:55.796067    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:09:58.021154 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.021110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:58.021567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.021163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:58.021735 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.021713    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:58.023731 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.023706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0855e8e-3df7-41a5-8d2d-eaad6ff17404-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ggsw\" (UID: \"d0855e8e-3df7-41a5-8d2d-eaad6ff17404\") " pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:58.127229 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.127193    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89spd\""
Sep 09 03:09:58.135933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.135910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ggsw"
Sep 09 03:09:58.433079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.433044    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ggsw"]
Sep 09 03:09:58.436469 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:09:58.436435    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0855e8e_3df7_41a5_8d2d_eaad6ff17404.slice/crio-521c54dd6347899487fb51966801bd4c3955dc120fe90b630457520059cd4ced WatchSource:0}: Error finding container 521c54dd6347899487fb51966801bd4c3955dc120fe90b630457520059cd4ced: Status 404 returned error can't find the container with id 521c54dd6347899487fb51966801bd4c3955dc120fe90b630457520059cd4ced
Sep 09 03:09:58.694994 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.694907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ggsw" event={"ID":"d0855e8e-3df7-41a5-8d2d-eaad6ff17404","Type":"ContainerStarted","Data":"423f3a23d5d187f7c956163274588ca1f5e66b7bbb52e39860f692f922402e53"}
Sep 09 03:09:58.694994 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:58.694966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ggsw" event={"ID":"d0855e8e-3df7-41a5-8d2d-eaad6ff17404","Type":"ContainerStarted","Data":"521c54dd6347899487fb51966801bd4c3955dc120fe90b630457520059cd4ced"}
Sep 09 03:09:59.699154 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:09:59.699123    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ggsw" event={"ID":"d0855e8e-3df7-41a5-8d2d-eaad6ff17404","Type":"ContainerStarted","Data":"876c65fc6d710676da2d3edda8ef5f997b18a9441f174967e4543e117ffe5662"}
Sep 09 03:10:01.708879 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:01.708839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ggsw" event={"ID":"d0855e8e-3df7-41a5-8d2d-eaad6ff17404","Type":"ContainerStarted","Data":"7594a587c9654a5f761ac7a09634388c474e871e47a7c7cfe8fe152fe98b2287"}
Sep 09 03:10:01.725398 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:01.725342    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2ggsw" podStartSLOduration=17.595583218 podStartE2EDuration="19.725324364s" totalImagesPullingTime="2.129741146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:09:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:01.724734989 +0000 UTC m=+250.287061558" watchObservedRunningTime="2026-09-09 03:10:01.725324364 +0000 UTC m=+250.287650931"
Sep 09 03:10:01.753739 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:01.753703    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:01.854352 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:01.854318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:01.970124 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:01.970034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:02.102282 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.102235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:02.203528 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.203492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:02.324785 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.324751    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:02.501239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.501201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:02.657714 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.657619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:10:02.660157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.660123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99b1fee-9619-4012-8bf1-19657cc1ac91-metrics-certs\") pod \"network-metrics-daemon-w75g9\" (UID: \"c99b1fee-9619-4012-8bf1-19657cc1ac91\") " pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:10:02.719599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.719563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:02.818824 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.818785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:02.835255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.835230    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bzb6q\""
Sep 09 03:10:02.843883 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.843865    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w75g9"
Sep 09 03:10:02.918687 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.918604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:02.976481 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:02.976446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:03.110027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:03.109991    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w75g9"]
Sep 09 03:10:03.112945 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:03.112917    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99b1fee_9619_4012_8bf1_19657cc1ac91.slice/crio-d94f6092c440c382aaa542949a97b3e88752a67d808905c8b0e4104cfc318645 WatchSource:0}: Error finding container d94f6092c440c382aaa542949a97b3e88752a67d808905c8b0e4104cfc318645: Status 404 returned error can't find the container with id d94f6092c440c382aaa542949a97b3e88752a67d808905c8b0e4104cfc318645
Sep 09 03:10:03.305345 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:03.305314    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:03.358189 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:03.358138    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:03.498830 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:03.498798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:03.716714 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:03.716627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w75g9" event={"ID":"c99b1fee-9619-4012-8bf1-19657cc1ac91","Type":"ContainerStarted","Data":"d94f6092c440c382aaa542949a97b3e88752a67d808905c8b0e4104cfc318645"}
Sep 09 03:10:03.891489 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:03.891448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:04.749989 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:04.749914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:04.850658 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:04.850612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:05.115163 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:05.115124    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:05.216242 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:05.216209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:05.726059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:05.726025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w75g9" event={"ID":"c99b1fee-9619-4012-8bf1-19657cc1ac91","Type":"ContainerStarted","Data":"f075118580443bc7534cd2c84274e58859df0664e3a008d5377b53b7c18421e4"}
Sep 09 03:10:05.726233 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:05.726077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w75g9" event={"ID":"c99b1fee-9619-4012-8bf1-19657cc1ac91","Type":"ContainerStarted","Data":"1ce4f2d1b7844ce997189c6d57967cdbf654bffa538c2ddfcbde5c2a21110267"}
Sep 09 03:10:05.741269 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:05.741225    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w75g9" podStartSLOduration=253.156129418 podStartE2EDuration="4m14.741208995s" totalImagesPullingTime="1.585079577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:05.739842076 +0000 UTC m=+254.302168634" watchObservedRunningTime="2026-09-09 03:10:05.741208995 +0000 UTC m=+254.303535539"
Sep 09 03:10:05.823847 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:05.823811    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:05.888908 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:05.888875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:06.682333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:06.682295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:06.747761 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:06.747718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:06.907248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:06.907209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:07.899068 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:07.899023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:10:07.901589 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:07.901566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4366ede7-855d-48f2-8285-e1fcf0e925ec-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4t4zd\" (UID: \"4366ede7-855d-48f2-8285-e1fcf0e925ec\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:10:07.974659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:07.974619    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:08.027367 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:08.027320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:08.090320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:08.090286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:08.162789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:08.162701    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d8fbz\""
Sep 09 03:10:08.171371 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:08.171345    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd"
Sep 09 03:10:08.439425 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:08.439337    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4t4zd"]
Sep 09 03:10:08.442186 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:08.442157    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4366ede7_855d_48f2_8285_e1fcf0e925ec.slice/crio-180ab093d24eb3d17db8c31c6ac8abf2ffb9fccb06bfc02782e099a3d04f3266 WatchSource:0}: Error finding container 180ab093d24eb3d17db8c31c6ac8abf2ffb9fccb06bfc02782e099a3d04f3266: Status 404 returned error can't find the container with id 180ab093d24eb3d17db8c31c6ac8abf2ffb9fccb06bfc02782e099a3d04f3266
Sep 09 03:10:08.720272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:08.720180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:08.739050 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:08.739014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd" event={"ID":"4366ede7-855d-48f2-8285-e1fcf0e925ec","Type":"ContainerStarted","Data":"180ab093d24eb3d17db8c31c6ac8abf2ffb9fccb06bfc02782e099a3d04f3266"}
Sep 09 03:10:09.744321 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:09.744235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd" event={"ID":"4366ede7-855d-48f2-8285-e1fcf0e925ec","Type":"ContainerStarted","Data":"5b3bbcefa68fdb869da3462a3e4c3ad84c3a1c3fe2a4f866c866a701d2a6d4f3"}
Sep 09 03:10:09.758758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:09.758701    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-4t4zd" podStartSLOduration=32.800185454 podStartE2EDuration="33.758679716s" totalImagesPullingTime="958.494262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:09.757667318 +0000 UTC m=+258.319993874" watchObservedRunningTime="2026-09-09 03:10:09.758679716 +0000 UTC m=+258.321006258"
Sep 09 03:10:09.853902 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:09.853856    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:09.914923 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:09.914800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:10.541184 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.541151    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79bf84b858-5r7wl"]
Sep 09 03:10:10.543270 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.543248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.546516 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.546490    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 03:10:10.546516 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.546511    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 03:10:10.546752 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.546528    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 03:10:10.546752 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.546576    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 03:10:10.547045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.547028    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 03:10:10.547325 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.547306    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 03:10:10.547438 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.547306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4b5tb\""
Sep 09 03:10:10.547438 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.547334    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 03:10:10.556528 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.556500    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bf84b858-5r7wl"]
Sep 09 03:10:10.627767 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.627721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-console-config\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.627969 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.627786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-oauth-serving-cert\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.627969 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.627807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-service-ca\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.627969 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.627826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-serving-cert\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.627969 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.627845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-oauth-config\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.627969 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.627859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgr6\" (UniqueName: \"kubernetes.io/projected/82600971-19fd-43c9-8230-e9365cb0eba7-kube-api-access-sjgr6\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.728570 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.728513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-service-ca\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.728570 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.728571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-serving-cert\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.728789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.728596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-oauth-config\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.728789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.728613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgr6\" (UniqueName: \"kubernetes.io/projected/82600971-19fd-43c9-8230-e9365cb0eba7-kube-api-access-sjgr6\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.728789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.728676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-console-config\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.728789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.728715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-oauth-serving-cert\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.729267 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.729241    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-oauth-serving-cert\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.729385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.729276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-service-ca\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.729385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.729307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-console-config\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.731781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.731759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-oauth-config\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.731781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.731779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-serving-cert\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.736855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.736833    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgr6\" (UniqueName: \"kubernetes.io/projected/82600971-19fd-43c9-8230-e9365cb0eba7-kube-api-access-sjgr6\") pod \"console-79bf84b858-5r7wl\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") " pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.839138 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.839056    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:10.853090 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.853053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:10.955729 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:10.955694    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:11.057732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.057694    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:11.076857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.076813    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"]
Sep 09 03:10:11.079036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.079017    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.081535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.081454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 03:10:11.081535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.081497    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 03:10:11.081535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.081509    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6plnc\""
Sep 09 03:10:11.081871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.081454    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 03:10:11.081871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.081454    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 03:10:11.081972 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.081905    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 03:10:11.091012 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.090941    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"]
Sep 09 03:10:11.123709 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.123663    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lrfff"]
Sep 09 03:10:11.126297 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.126275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.128701 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.128669    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 03:10:11.128842 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.128824    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 03:10:11.128920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.128906    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tzldt\""
Sep 09 03:10:11.129071 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.129053    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 03:10:11.132997 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.132965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/877cf394-5003-4c55-89de-088237996d57-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.133130 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.133015    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/877cf394-5003-4c55-89de-088237996d57-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.133130 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.133101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52p4s\" (UniqueName: \"kubernetes.io/projected/877cf394-5003-4c55-89de-088237996d57-kube-api-access-52p4s\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.133366 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.133163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/877cf394-5003-4c55-89de-088237996d57-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.136703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.136677    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bf84b858-5r7wl"]
Sep 09 03:10:11.142002 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:11.141966    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82600971_19fd_43c9_8230_e9365cb0eba7.slice/crio-254c54ceece6f460d677c39fd9f4c6bb4e20a343f09c28939840e87b505abfcf WatchSource:0}: Error finding container 254c54ceece6f460d677c39fd9f4c6bb4e20a343f09c28939840e87b505abfcf: Status 404 returned error can't find the container with id 254c54ceece6f460d677c39fd9f4c6bb4e20a343f09c28939840e87b505abfcf
Sep 09 03:10:11.160767 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.160736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:11.214131 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.214079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:11.233710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/877cf394-5003-4c55-89de-088237996d57-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.233882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-tls\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.233882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/877cf394-5003-4c55-89de-088237996d57-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.233882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-sys\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.233882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwr4z\" (UniqueName: \"kubernetes.io/projected/9e7661a2-2851-40c7-8f24-f7901b78847b-kube-api-access-cwr4z\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.234082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52p4s\" (UniqueName: \"kubernetes.io/projected/877cf394-5003-4c55-89de-088237996d57-kube-api-access-52p4s\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.234082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-root\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.234082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-wtmp\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.234082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.233989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/877cf394-5003-4c55-89de-088237996d57-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.234082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.234065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e7661a2-2851-40c7-8f24-f7901b78847b-metrics-client-ca\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.234301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.234119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.234301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.234152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-textfile\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.234301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.234187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.234452 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.234424    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/877cf394-5003-4c55-89de-088237996d57-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.236722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.236699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/877cf394-5003-4c55-89de-088237996d57-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.236814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.236768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/877cf394-5003-4c55-89de-088237996d57-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.241509 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.241489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52p4s\" (UniqueName: \"kubernetes.io/projected/877cf394-5003-4c55-89de-088237996d57-kube-api-access-52p4s\") pod \"openshift-state-metrics-657fd6c54-58dll\" (UID: \"877cf394-5003-4c55-89de-088237996d57\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.315242 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.315197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:11.335225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-tls\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-sys\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwr4z\" (UniqueName: \"kubernetes.io/projected/9e7661a2-2851-40c7-8f24-f7901b78847b-kube-api-access-cwr4z\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-root\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-wtmp\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-sys\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e7661a2-2851-40c7-8f24-f7901b78847b-metrics-client-ca\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335419 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:10:11.335364    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 03:10:11.335419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-root\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335810 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:10:11.335433    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-tls podName:9e7661a2-2851-40c7-8f24-f7901b78847b nodeName:}" failed. No retries permitted until 2026-09-09 03:10:11.835411974 +0000 UTC m=+260.397738504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-tls") pod "node-exporter-lrfff" (UID: "9e7661a2-2851-40c7-8f24-f7901b78847b") : secret "node-exporter-tls" not found
Sep 09 03:10:11.335810 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-textfile\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335810 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-wtmp\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.335810 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.336005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.335847    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-textfile\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.336057 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.336044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.336112 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.336044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e7661a2-2851-40c7-8f24-f7901b78847b-metrics-client-ca\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.338078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.338049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.342737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.342673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwr4z\" (UniqueName: \"kubernetes.io/projected/9e7661a2-2851-40c7-8f24-f7901b78847b-kube-api-access-cwr4z\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.389880 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.389850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"
Sep 09 03:10:11.421050 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.421013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:11.475239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.475187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:11.662436 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.662325    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-58dll"]
Sep 09 03:10:11.664200 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:11.664169    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877cf394_5003_4c55_89de_088237996d57.slice/crio-6de2c3922da8e2928c09a0bafcf3543726a5894f6401df2f0466242478fee563 WatchSource:0}: Error finding container 6de2c3922da8e2928c09a0bafcf3543726a5894f6401df2f0466242478fee563: Status 404 returned error can't find the container with id 6de2c3922da8e2928c09a0bafcf3543726a5894f6401df2f0466242478fee563
Sep 09 03:10:11.707686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.707587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:11.754081 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.754045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll" event={"ID":"877cf394-5003-4c55-89de-088237996d57","Type":"ContainerStarted","Data":"2caabb699378ce9118b462259facd237ff31b8a70a4634f86dee66fd89f0c1ad"}
Sep 09 03:10:11.754304 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.754122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll" event={"ID":"877cf394-5003-4c55-89de-088237996d57","Type":"ContainerStarted","Data":"6de2c3922da8e2928c09a0bafcf3543726a5894f6401df2f0466242478fee563"}
Sep 09 03:10:11.755816 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.755747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bf84b858-5r7wl" event={"ID":"82600971-19fd-43c9-8230-e9365cb0eba7","Type":"ContainerStarted","Data":"254c54ceece6f460d677c39fd9f4c6bb4e20a343f09c28939840e87b505abfcf"}
Sep 09 03:10:11.840251 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.840218    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-tls\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:11.843011 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:11.842973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e7661a2-2851-40c7-8f24-f7901b78847b-node-exporter-tls\") pod \"node-exporter-lrfff\" (UID: \"9e7661a2-2851-40c7-8f24-f7901b78847b\") " pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:12.037032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:12.036990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lrfff"
Sep 09 03:10:12.053733 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:12.053644    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:12.056607 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:12.056577    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7661a2_2851_40c7_8f24_f7901b78847b.slice/crio-24f9158f3feb2f96749f5b3bff114751b0a1dbfa3509ed7c5a2575ff9df145a3 WatchSource:0}: Error finding container 24f9158f3feb2f96749f5b3bff114751b0a1dbfa3509ed7c5a2575ff9df145a3: Status 404 returned error can't find the container with id 24f9158f3feb2f96749f5b3bff114751b0a1dbfa3509ed7c5a2575ff9df145a3
Sep 09 03:10:12.132561 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:12.132503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:12.763589 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:12.763436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrfff" event={"ID":"9e7661a2-2851-40c7-8f24-f7901b78847b","Type":"ContainerStarted","Data":"24f9158f3feb2f96749f5b3bff114751b0a1dbfa3509ed7c5a2575ff9df145a3"}
Sep 09 03:10:12.766192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:12.766123    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll" event={"ID":"877cf394-5003-4c55-89de-088237996d57","Type":"ContainerStarted","Data":"575a0e6cee8f747f3907e2c880140cc89ae5c0236b623fb75d4c21afa58569ba"}
Sep 09 03:10:12.888485 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:12.888267    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:13.058809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.058770    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-845787b854-c79nh"]
Sep 09 03:10:13.063617 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.063590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.065981 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.065949    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m72lc\""
Sep 09 03:10:13.065981 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.065976    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 03:10:13.066191 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.065976    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 03:10:13.066191 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.066099    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 03:10:13.066191 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.065949    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 03:10:13.066470 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.066454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 03:10:13.066526 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.066454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dq4817nqs3lmh\""
Sep 09 03:10:13.073444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.073416    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-845787b854-c79nh"]
Sep 09 03:10:13.151931 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.151893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.151931 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.151946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.152154 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.152020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9f4522a-4fda-4785-9d24-0657ef72835b-metrics-client-ca\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.152154 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.152101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.152154 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.152147    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.152303 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.152167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-grpc-tls\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.152303 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.152202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrmg\" (UniqueName: \"kubernetes.io/projected/c9f4522a-4fda-4785-9d24-0657ef72835b-kube-api-access-7lrmg\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.152303 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.152289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-tls\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.253176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.253135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9f4522a-4fda-4785-9d24-0657ef72835b-metrics-client-ca\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.253368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.253200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.253368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.253233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.253368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.253253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-grpc-tls\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.253368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.253270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrmg\" (UniqueName: \"kubernetes.io/projected/c9f4522a-4fda-4785-9d24-0657ef72835b-kube-api-access-7lrmg\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.253368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.253318    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-tls\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.253368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.253348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.253757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.253379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.254279 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.254048    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9f4522a-4fda-4785-9d24-0657ef72835b-metrics-client-ca\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.256845 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.256821    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-tls\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.257290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.257243    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.257376 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.257320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.257425 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.257396    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.257557 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.257519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-grpc-tls\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.257630 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.257587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9f4522a-4fda-4785-9d24-0657ef72835b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.261718 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.261697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrmg\" (UniqueName: \"kubernetes.io/projected/c9f4522a-4fda-4785-9d24-0657ef72835b-kube-api-access-7lrmg\") pod \"thanos-querier-845787b854-c79nh\" (UID: \"c9f4522a-4fda-4785-9d24-0657ef72835b\") " pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:13.376974 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:13.376882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:14.011439 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.011402    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:14.065804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.065713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:14.260724 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.260691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:14.396515 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.396475    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:14.486403 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.486369    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-845787b854-c79nh"]
Sep 09 03:10:14.489796 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:14.489761    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f4522a_4fda_4785_9d24_0657ef72835b.slice/crio-10a891dc101c6a66c25068a5c2bce99408cf670ec0d79df41300bd65331a5103 WatchSource:0}: Error finding container 10a891dc101c6a66c25068a5c2bce99408cf670ec0d79df41300bd65331a5103: Status 404 returned error can't find the container with id 10a891dc101c6a66c25068a5c2bce99408cf670ec0d79df41300bd65331a5103
Sep 09 03:10:14.775383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.775337    2562 generic.go:356] "Generic (PLEG): container finished" podID="9e7661a2-2851-40c7-8f24-f7901b78847b" containerID="d6183a8b83a5c8ef013dfe9d4794baf93b09f1bcbe0177855a9d8d2bb259d2c0" exitCode=0
Sep 09 03:10:14.775603 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.775422    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrfff" event={"ID":"9e7661a2-2851-40c7-8f24-f7901b78847b","Type":"ContainerDied","Data":"d6183a8b83a5c8ef013dfe9d4794baf93b09f1bcbe0177855a9d8d2bb259d2c0"}
Sep 09 03:10:14.777686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.777657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll" event={"ID":"877cf394-5003-4c55-89de-088237996d57","Type":"ContainerStarted","Data":"1545b8a66cfa903be164ab9adf3d7380a895305067c04151d6d20a8f0abf034d"}
Sep 09 03:10:14.779348 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.779320    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bf84b858-5r7wl" event={"ID":"82600971-19fd-43c9-8230-e9365cb0eba7","Type":"ContainerStarted","Data":"5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18"}
Sep 09 03:10:14.780654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.780634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-845787b854-c79nh" event={"ID":"c9f4522a-4fda-4785-9d24-0657ef72835b","Type":"ContainerStarted","Data":"10a891dc101c6a66c25068a5c2bce99408cf670ec0d79df41300bd65331a5103"}
Sep 09 03:10:14.806385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.806332    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79bf84b858-5r7wl" podStartSLOduration=1.725338459 podStartE2EDuration="4.806313854s" totalImagesPullingTime="3.080975395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:14.80469356 +0000 UTC m=+263.367020105" watchObservedRunningTime="2026-09-09 03:10:14.806313854 +0000 UTC m=+263.368640398"
Sep 09 03:10:14.820644 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:14.820594    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-58dll" podStartSLOduration=1.413671325 podStartE2EDuration="3.820575107s" totalImagesPullingTime="2.406903782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:14.819070441 +0000 UTC m=+263.381396987" watchObservedRunningTime="2026-09-09 03:10:14.820575107 +0000 UTC m=+263.382901651"
Sep 09 03:10:15.786753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:15.786705    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrfff" event={"ID":"9e7661a2-2851-40c7-8f24-f7901b78847b","Type":"ContainerStarted","Data":"7b1d3450bc69fdca1da10d1811e060ff48067c341d9f0b119bda8d8648dbf582"}
Sep 09 03:10:15.787163 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:15.786778    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrfff" event={"ID":"9e7661a2-2851-40c7-8f24-f7901b78847b","Type":"ContainerStarted","Data":"faeea1f5c40c1e24490416cced948f8312e663885c51f6337866d877f1772f35"}
Sep 09 03:10:15.805408 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:15.805353    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lrfff" podStartSLOduration=2.567699456 podStartE2EDuration="4.805334381s" totalImagesPullingTime="2.153967388s" totalInitContainerRuntime="83.667537ms" isStatefulPod=true podCreationTimestamp="2026-09-09 03:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:15.805288853 +0000 UTC m=+264.367615399" watchObservedRunningTime="2026-09-09 03:10:15.805334381 +0000 UTC m=+264.367660918"
Sep 09 03:10:15.846343 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:15.846290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:15.914986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:15.914938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:16.660807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:16.660772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:16.783011 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:16.782915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:16.792900 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:16.792871    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-845787b854-c79nh" event={"ID":"c9f4522a-4fda-4785-9d24-0657ef72835b","Type":"ContainerStarted","Data":"7c249b9383e22fd63c73426f941846d83137603ea988fe936822a0cf6dec320c"}
Sep 09 03:10:16.883398 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:16.883366    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:17.292197 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:17.292156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:17.586658 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:17.586532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:17.800626 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:17.800582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-845787b854-c79nh" event={"ID":"c9f4522a-4fda-4785-9d24-0657ef72835b","Type":"ContainerStarted","Data":"4a238cd0f67082a65bf523bf9c2a97df5d3a432e6ccf21ecbff6052eb66592e5"}
Sep 09 03:10:17.801033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:17.800662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-845787b854-c79nh" event={"ID":"c9f4522a-4fda-4785-9d24-0657ef72835b","Type":"ContainerStarted","Data":"effc66355ead4fcb2ebeef1c82509f8c5f53c05653ca09c7b50fb4bc9ea15980"}
Sep 09 03:10:18.808345 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:18.808303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-845787b854-c79nh" event={"ID":"c9f4522a-4fda-4785-9d24-0657ef72835b","Type":"ContainerStarted","Data":"074bdeb7daf2c9fe601ec0bdd95a5d2d761af72e8eb86b14b178a37f6e713f80"}
Sep 09 03:10:18.808850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:18.808425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-845787b854-c79nh" event={"ID":"c9f4522a-4fda-4785-9d24-0657ef72835b","Type":"ContainerStarted","Data":"a8373756875c39b99ba23db8b689be2590fd001c8df543d3a8f6dce024b25ccb"}
Sep 09 03:10:18.808850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:18.808478    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:18.808850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:18.808532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-845787b854-c79nh" event={"ID":"c9f4522a-4fda-4785-9d24-0657ef72835b","Type":"ContainerStarted","Data":"c703b657631049d3acc9726d2e959a36ad44edafee0a2e91438e41d6511cb1b0"}
Sep 09 03:10:18.830518 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:18.830454    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-845787b854-c79nh" podStartSLOduration=2.712733262 podStartE2EDuration="5.830434876s" totalImagesPullingTime="3.117701614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:18.829191861 +0000 UTC m=+267.391518407" watchObservedRunningTime="2026-09-09 03:10:18.830434876 +0000 UTC m=+267.392761418"
Sep 09 03:10:20.853341 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:20.853300    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:20.853874 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:20.853366    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:20.854824 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:20.854796    2562 patch_prober.go:28] interesting pod/console-79bf84b858-5r7wl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body=
Sep 09 03:10:20.854937 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:20.854846    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-79bf84b858-5r7wl" podUID="82600971-19fd-43c9-8230-e9365cb0eba7" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused"
Sep 09 03:10:21.520157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:21.520107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:21.587233 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:21.587201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:23.103938 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:23.103900    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:23.153884 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:23.153842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:23.328009 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:23.327974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:24.818322 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:24.818292    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-845787b854-c79nh"
Sep 09 03:10:28.086704 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.086666    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bf84b858-5r7wl"]
Sep 09 03:10:28.109996 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.109966    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-698c548864-9gfcv"]
Sep 09 03:10:28.114412 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.114393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.121364 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.121336    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 03:10:28.121676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.121652    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698c548864-9gfcv"]
Sep 09 03:10:28.190313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.190273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6pl\" (UniqueName: \"kubernetes.io/projected/e89852b3-e46d-4a48-8d0c-8a00823ee79c-kube-api-access-ss6pl\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.190470 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.190443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-oauth-config\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.190560 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.190481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-config\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.190560 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.190507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-service-ca\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.190560 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.190552    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-trusted-ca-bundle\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.190714 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.190577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-serving-cert\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.190714 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.190643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-oauth-serving-cert\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.291822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.291780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-oauth-config\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.291822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.291822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-config\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.291822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.291843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-service-ca\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.292126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.291962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-trusted-ca-bundle\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.292126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.291999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-serving-cert\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.292126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.292038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-oauth-serving-cert\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.292268 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.292159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6pl\" (UniqueName: \"kubernetes.io/projected/e89852b3-e46d-4a48-8d0c-8a00823ee79c-kube-api-access-ss6pl\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.292611 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.292588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-config\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.292716 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.292631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-service-ca\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.292885 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.292861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-oauth-serving-cert\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.293006 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.292988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-trusted-ca-bundle\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.295073 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.295049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-oauth-config\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.295257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.295237    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-serving-cert\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.300083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.300065    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6pl\" (UniqueName: \"kubernetes.io/projected/e89852b3-e46d-4a48-8d0c-8a00823ee79c-kube-api-access-ss6pl\") pod \"console-698c548864-9gfcv\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") " pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.424803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.424708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:28.698408 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.698388    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698c548864-9gfcv"]
Sep 09 03:10:28.700899 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:28.700867    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89852b3_e46d_4a48_8d0c_8a00823ee79c.slice/crio-44623fe99f7918482018d09e8ab6a21c2b61ac6c05bb4de7a78bb51f92d62c4a WatchSource:0}: Error finding container 44623fe99f7918482018d09e8ab6a21c2b61ac6c05bb4de7a78bb51f92d62c4a: Status 404 returned error can't find the container with id 44623fe99f7918482018d09e8ab6a21c2b61ac6c05bb4de7a78bb51f92d62c4a
Sep 09 03:10:28.846558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.846511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698c548864-9gfcv" event={"ID":"e89852b3-e46d-4a48-8d0c-8a00823ee79c","Type":"ContainerStarted","Data":"b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed"}
Sep 09 03:10:28.846806 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.846578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698c548864-9gfcv" event={"ID":"e89852b3-e46d-4a48-8d0c-8a00823ee79c","Type":"ContainerStarted","Data":"44623fe99f7918482018d09e8ab6a21c2b61ac6c05bb4de7a78bb51f92d62c4a"}
Sep 09 03:10:28.863632 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:28.863583    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-698c548864-9gfcv" podStartSLOduration=0.863567538 podStartE2EDuration="863.567538ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:28.862471446 +0000 UTC m=+277.424797992" watchObservedRunningTime="2026-09-09 03:10:28.863567538 +0000 UTC m=+277.425894088"
Sep 09 03:10:31.524449 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:31.524406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:31.624614 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:31.624573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:32.441576 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:10:32.441519    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" podUID="79426f9f-2ce5-411d-85f5-3f74010b0ac6"
Sep 09 03:10:32.441724 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:10:32.441519    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d466q" podUID="bbb59c21-963b-4254-8d19-d2fb97b78882"
Sep 09 03:10:32.862500 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:32.862466    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:10:32.862893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:32.862469    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d466q"
Sep 09 03:10:35.754455 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:35.754407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:10:35.754912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:35.754507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:10:35.756980 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:35.756945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb59c21-963b-4254-8d19-d2fb97b78882-metrics-tls\") pod \"dns-default-d466q\" (UID: \"bbb59c21-963b-4254-8d19-d2fb97b78882\") " pod="openshift-dns/dns-default-d466q"
Sep 09 03:10:35.757123 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:35.757103    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"image-registry-56749b4d77-d7p5w\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") " pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:10:35.866027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:35.865993    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nklpr\""
Sep 09 03:10:35.866291 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:35.866275    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8s7tp\""
Sep 09 03:10:35.874333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:35.874303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d466q"
Sep 09 03:10:35.874451 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:35.874395    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:10:36.156306 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:36.156270    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d466q"]
Sep 09 03:10:36.158807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:36.158783    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56749b4d77-d7p5w"]
Sep 09 03:10:36.160190 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:36.160163    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb59c21_963b_4254_8d19_d2fb97b78882.slice/crio-553fef17bb90f06c83a71c383a0f7bad5ba5483aa754c49543388be0882b5938 WatchSource:0}: Error finding container 553fef17bb90f06c83a71c383a0f7bad5ba5483aa754c49543388be0882b5938: Status 404 returned error can't find the container with id 553fef17bb90f06c83a71c383a0f7bad5ba5483aa754c49543388be0882b5938
Sep 09 03:10:36.161064 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:10:36.161027    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79426f9f_2ce5_411d_85f5_3f74010b0ac6.slice/crio-349eda69d8400c09934c02be47dd9a3e2f8165264d3e8e5f50d50a2940e12f30 WatchSource:0}: Error finding container 349eda69d8400c09934c02be47dd9a3e2f8165264d3e8e5f50d50a2940e12f30: Status 404 returned error can't find the container with id 349eda69d8400c09934c02be47dd9a3e2f8165264d3e8e5f50d50a2940e12f30
Sep 09 03:10:36.879163 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:36.879113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" event={"ID":"79426f9f-2ce5-411d-85f5-3f74010b0ac6","Type":"ContainerStarted","Data":"f40d16cd16a97a7456f4247bcd0b159a8f29f3e65e1d457f0a39a914c410765f"}
Sep 09 03:10:36.879681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:36.879191    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:10:36.879681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:36.879215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" event={"ID":"79426f9f-2ce5-411d-85f5-3f74010b0ac6","Type":"ContainerStarted","Data":"349eda69d8400c09934c02be47dd9a3e2f8165264d3e8e5f50d50a2940e12f30"}
Sep 09 03:10:36.880592 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:36.880563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d466q" event={"ID":"bbb59c21-963b-4254-8d19-d2fb97b78882","Type":"ContainerStarted","Data":"553fef17bb90f06c83a71c383a0f7bad5ba5483aa754c49543388be0882b5938"}
Sep 09 03:10:36.898278 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:36.898225    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" podStartSLOduration=284.898204971 podStartE2EDuration="4m44.898204971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:36.897821798 +0000 UTC m=+285.460148346" watchObservedRunningTime="2026-09-09 03:10:36.898204971 +0000 UTC m=+285.460531551"
Sep 09 03:10:38.425240 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:38.425201    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:38.425638 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:38.425276    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:38.426705 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:38.426679    2562 patch_prober.go:28] interesting pod/console-698c548864-9gfcv container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.19:8443/health\": dial tcp 10.133.0.19:8443: connect: connection refused" start-of-body=
Sep 09 03:10:38.426757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:38.426728    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-698c548864-9gfcv" podUID="e89852b3-e46d-4a48-8d0c-8a00823ee79c" containerName="console" probeResult="failure" output="Get \"https://10.133.0.19:8443/health\": dial tcp 10.133.0.19:8443: connect: connection refused"
Sep 09 03:10:38.891192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:38.891157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d466q" event={"ID":"bbb59c21-963b-4254-8d19-d2fb97b78882","Type":"ContainerStarted","Data":"51dae9e77d521407462355c72a6e9dac201914c869235d413ddbef12b7c65ee6"}
Sep 09 03:10:38.891336 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:38.891235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d466q" event={"ID":"bbb59c21-963b-4254-8d19-d2fb97b78882","Type":"ContainerStarted","Data":"fba40d9cef0ecdd5542b1862c513530532ca87cb534165c708243824ba54fe9c"}
Sep 09 03:10:38.891336 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:38.891303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d466q"
Sep 09 03:10:38.907895 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:38.907826    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d466q" podStartSLOduration=252.226646303 podStartE2EDuration="4m13.907808434s" totalImagesPullingTime="1.681162131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:38.905639732 +0000 UTC m=+287.467966277" watchObservedRunningTime="2026-09-09 03:10:38.907808434 +0000 UTC m=+287.470134978"
Sep 09 03:10:39.408161 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:39.408124    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:41.037710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:41.037669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:41.137887 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:41.137850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:46.780142 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:46.780082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:46.834200 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:46.834151    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:48.425161 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:48.425116    2562 patch_prober.go:28] interesting pod/console-698c548864-9gfcv container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.19:8443/health\": dial tcp 10.133.0.19:8443: connect: connection refused" start-of-body=
Sep 09 03:10:48.425568 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:48.425172    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-698c548864-9gfcv" podUID="e89852b3-e46d-4a48-8d0c-8a00823ee79c" containerName="console" probeResult="failure" output="Get \"https://10.133.0.19:8443/health\": dial tcp 10.133.0.19:8443: connect: connection refused"
Sep 09 03:10:48.897340 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:48.897302    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d466q"
Sep 09 03:10:50.655827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:50.655795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:50.707746 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:50.707697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:51.807657 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:51.807629    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 03:10:53.108147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.108096    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79bf84b858-5r7wl" podUID="82600971-19fd-43c9-8230-e9365cb0eba7" containerName="console" containerID="cri-o://5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18" gracePeriod=15
Sep 09 03:10:53.355992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.355967    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:53.386115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386035    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-oauth-serving-cert\") pod \"82600971-19fd-43c9-8230-e9365cb0eba7\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") "
Sep 09 03:10:53.386115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386088    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-oauth-config\") pod \"82600971-19fd-43c9-8230-e9365cb0eba7\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") "
Sep 09 03:10:53.386115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386111    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjgr6\" (UniqueName: \"kubernetes.io/projected/82600971-19fd-43c9-8230-e9365cb0eba7-kube-api-access-sjgr6\") pod \"82600971-19fd-43c9-8230-e9365cb0eba7\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") "
Sep 09 03:10:53.386372 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-serving-cert\") pod \"82600971-19fd-43c9-8230-e9365cb0eba7\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") "
Sep 09 03:10:53.386372 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386172    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-console-config\") pod \"82600971-19fd-43c9-8230-e9365cb0eba7\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") "
Sep 09 03:10:53.386372 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386212    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-service-ca\") pod \"82600971-19fd-43c9-8230-e9365cb0eba7\" (UID: \"82600971-19fd-43c9-8230-e9365cb0eba7\") "
Sep 09 03:10:53.386576 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "82600971-19fd-43c9-8230-e9365cb0eba7" (UID: "82600971-19fd-43c9-8230-e9365cb0eba7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:10:53.386731 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386700    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-console-config" (OuterVolumeSpecName: "console-config") pod "82600971-19fd-43c9-8230-e9365cb0eba7" (UID: "82600971-19fd-43c9-8230-e9365cb0eba7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:10:53.386778 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.386727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-service-ca" (OuterVolumeSpecName: "service-ca") pod "82600971-19fd-43c9-8230-e9365cb0eba7" (UID: "82600971-19fd-43c9-8230-e9365cb0eba7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:10:53.388837 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.388799    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "82600971-19fd-43c9-8230-e9365cb0eba7" (UID: "82600971-19fd-43c9-8230-e9365cb0eba7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:10:53.389083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.389055    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "82600971-19fd-43c9-8230-e9365cb0eba7" (UID: "82600971-19fd-43c9-8230-e9365cb0eba7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:10:53.389083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.389068    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82600971-19fd-43c9-8230-e9365cb0eba7-kube-api-access-sjgr6" (OuterVolumeSpecName: "kube-api-access-sjgr6") pod "82600971-19fd-43c9-8230-e9365cb0eba7" (UID: "82600971-19fd-43c9-8230-e9365cb0eba7"). InnerVolumeSpecName "kube-api-access-sjgr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:10:53.487669 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.487615    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-console-config\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:10:53.487669 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.487660    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:10:53.487669 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.487669    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82600971-19fd-43c9-8230-e9365cb0eba7-oauth-serving-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:10:53.487669 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.487679    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-oauth-config\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:10:53.487669 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.487689    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgr6\" (UniqueName: \"kubernetes.io/projected/82600971-19fd-43c9-8230-e9365cb0eba7-kube-api-access-sjgr6\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:10:53.487973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.487698    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82600971-19fd-43c9-8230-e9365cb0eba7-console-serving-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:10:53.951397 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.951365    2562 generic.go:356] "Generic (PLEG): container finished" podID="82600971-19fd-43c9-8230-e9365cb0eba7" containerID="5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18" exitCode=0
Sep 09 03:10:53.951581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.951465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bf84b858-5r7wl" event={"ID":"82600971-19fd-43c9-8230-e9365cb0eba7","Type":"ContainerDied","Data":"5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18"}
Sep 09 03:10:53.951581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.951477    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bf84b858-5r7wl"
Sep 09 03:10:53.951581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.951531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bf84b858-5r7wl" event={"ID":"82600971-19fd-43c9-8230-e9365cb0eba7","Type":"ContainerDied","Data":"254c54ceece6f460d677c39fd9f4c6bb4e20a343f09c28939840e87b505abfcf"}
Sep 09 03:10:53.951737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.951576    2562 scope.go:117] "RemoveContainer" containerID="5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18"
Sep 09 03:10:53.960254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.960234    2562 scope.go:117] "RemoveContainer" containerID="5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18"
Sep 09 03:10:53.960605 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:10:53.960581    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18\": container with ID starting with 5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18 not found: ID does not exist" containerID="5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18"
Sep 09 03:10:53.960688 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.960617    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18"} err="failed to get container status \"5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18\": rpc error: code = NotFound desc = could not find container \"5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18\": container with ID starting with 5a0969ffead4238a4938392dcf368d4a75f9f879d22ef8f6564dc674f1431c18 not found: ID does not exist"
Sep 09 03:10:53.965922 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.965891    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bf84b858-5r7wl"]
Sep 09 03:10:53.969085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:53.969057    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79bf84b858-5r7wl"]
Sep 09 03:10:55.878964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:55.878924    2562 patch_prober.go:28] interesting pod/image-registry-56749b4d77-d7p5w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:55.879371 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:55.878994    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" podUID="79426f9f-2ce5-411d-85f5-3f74010b0ac6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:55.937760 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:55.937724    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82600971-19fd-43c9-8230-e9365cb0eba7" path="/var/lib/kubelet/pods/82600971-19fd-43c9-8230-e9365cb0eba7/volumes"
Sep 09 03:10:57.784066 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:57.784036    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:57.884915 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:57.884872    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:57.890674 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:57.890649    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:10:58.009347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:58.009315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:58.158652 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:58.158528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:58.259931 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:58.259882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:58.429251 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:58.429159    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:58.433201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:58.433171    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:10:58.588831 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:58.588793    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:58.685694 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:58.685596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:58.852109 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:58.852071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:59.006725 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:59.006621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:59.061008 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:59.060974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:10:59.982139 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:59.982103    2562 generic.go:356] "Generic (PLEG): container finished" podID="c34dac72-7fef-47aa-889f-67f866f4f1e6" containerID="9de227aefef083e15a3c31b591c29e510eec1d437ab346ab47206b29a1c7f37b" exitCode=0
Sep 09 03:10:59.982590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:59.982159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422" event={"ID":"c34dac72-7fef-47aa-889f-67f866f4f1e6","Type":"ContainerDied","Data":"9de227aefef083e15a3c31b591c29e510eec1d437ab346ab47206b29a1c7f37b"}
Sep 09 03:10:59.982590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:10:59.982466    2562 scope.go:117] "RemoveContainer" containerID="9de227aefef083e15a3c31b591c29e510eec1d437ab346ab47206b29a1c7f37b"
Sep 09 03:11:00.995623 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:00.995573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j5422" event={"ID":"c34dac72-7fef-47aa-889f-67f866f4f1e6","Type":"ContainerStarted","Data":"3b349affcd01cfead5ec79306feb9994ebaf4955b57b375cd6a50f086632157f"}
Sep 09 03:11:02.866528 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:02.866493    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56749b4d77-d7p5w"]
Sep 09 03:11:02.941588 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:02.941520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:03.057643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:03.057598    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:03.159102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:03.159011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:07.114941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:07.114900    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:07.168505 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:07.168470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:09.354467 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:09.354429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:09.455730 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:09.455693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:13.311029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:13.310997    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56749b4d77-d7p5w_79426f9f-2ce5-411d-85f5-3f74010b0ac6/registry/0.log"
Sep 09 03:11:13.910980 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:13.910927    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwtvs_781959c0-527d-4192-86a4-99be89ec07ab/node-ca/0.log"
Sep 09 03:11:14.111083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:14.111052    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-794f5b986b-77gpk_7042025b-b615-4b6e-a06c-eda170b933bb/router/0.log"
Sep 09 03:11:14.311115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:14.311087    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6n2dq_be59ed35-76ae-464d-b7ff-5c3ded0d76c5/serve-healthcheck-canary/0.log"
Sep 09 03:11:16.780512 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:16.780482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:16.833436 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:16.833384    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:22.740495 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:22.740452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:22.841255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:22.841223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:22.894614 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:22.894576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:27.206030 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.205989    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:27.206443 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.206203    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c754958-dw2kp"]
Sep 09 03:11:27.206532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.206506    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82600971-19fd-43c9-8230-e9365cb0eba7" containerName="console"
Sep 09 03:11:27.206532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.206517    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="82600971-19fd-43c9-8230-e9365cb0eba7" containerName="console"
Sep 09 03:11:27.206631 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.206599    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="82600971-19fd-43c9-8230-e9365cb0eba7" containerName="console"
Sep 09 03:11:27.209307 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.209286    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.222305 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.222275    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c754958-dw2kp"]
Sep 09 03:11:27.289773 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.289731    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:27.367708 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.367667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-oauth-serving-cert\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.367708 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.367709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-oauth-config\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.367930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.367757    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-serving-cert\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.367930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.367775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-service-ca\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.367930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.367794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-trusted-ca-bundle\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.367930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.367811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xft9m\" (UniqueName: \"kubernetes.io/projected/7feebf7b-e074-4367-9994-018df06bf800-kube-api-access-xft9m\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.367930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.367862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-console-config\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.391240 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.391199    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:27.469286 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.469172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-oauth-serving-cert\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.469286 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.469221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-oauth-config\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.469286 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.469262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-serving-cert\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.469286 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.469287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-service-ca\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.469686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.469313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-trusted-ca-bundle\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.469686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.469338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xft9m\" (UniqueName: \"kubernetes.io/projected/7feebf7b-e074-4367-9994-018df06bf800-kube-api-access-xft9m\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.469686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.469362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-console-config\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.470234 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.470203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-service-ca\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.470360 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.470209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-console-config\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.470431 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.470358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-oauth-serving-cert\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.470431 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.470372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-trusted-ca-bundle\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.472151 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.472127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-oauth-config\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.472374 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.472357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-serving-cert\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.477232 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.477210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xft9m\" (UniqueName: \"kubernetes.io/projected/7feebf7b-e074-4367-9994-018df06bf800-kube-api-access-xft9m\") pod \"console-78c754958-dw2kp\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") " pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.491477 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.491439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:27.518469 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.518424    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:27.539224 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.539180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:27.792891 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.792868    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c754958-dw2kp"]
Sep 09 03:11:27.795849 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:11:27.795817    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7feebf7b_e074_4367_9994_018df06bf800.slice/crio-d326b741aba15dc2a5a132cc05143e0743291985d7c31ed54acc6f627bc5ac8b WatchSource:0}: Error finding container d326b741aba15dc2a5a132cc05143e0743291985d7c31ed54acc6f627bc5ac8b: Status 404 returned error can't find the container with id d326b741aba15dc2a5a132cc05143e0743291985d7c31ed54acc6f627bc5ac8b
Sep 09 03:11:27.887320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:27.887264    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" podUID="79426f9f-2ce5-411d-85f5-3f74010b0ac6" containerName="registry" containerID="cri-o://f40d16cd16a97a7456f4247bcd0b159a8f29f3e65e1d457f0a39a914c410765f" gracePeriod=30
Sep 09 03:11:28.003955 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:28.003915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:28.054835 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:28.054750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:28.108074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:28.108041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c754958-dw2kp" event={"ID":"7feebf7b-e074-4367-9994-018df06bf800","Type":"ContainerStarted","Data":"0364719346c88c68fe5b9f152e3c7128fc7901e32fca1e4fc13aaadd76a93608"}
Sep 09 03:11:28.108274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:28.108090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c754958-dw2kp" event={"ID":"7feebf7b-e074-4367-9994-018df06bf800","Type":"ContainerStarted","Data":"d326b741aba15dc2a5a132cc05143e0743291985d7c31ed54acc6f627bc5ac8b"}
Sep 09 03:11:28.123814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:28.123750    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78c754958-dw2kp" podStartSLOduration=1.123730037 podStartE2EDuration="1.123730037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:11:28.123313246 +0000 UTC m=+336.685639791" watchObservedRunningTime="2026-09-09 03:11:28.123730037 +0000 UTC m=+336.686056593"
Sep 09 03:11:29.114062 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.114028    2562 generic.go:356] "Generic (PLEG): container finished" podID="79426f9f-2ce5-411d-85f5-3f74010b0ac6" containerID="f40d16cd16a97a7456f4247bcd0b159a8f29f3e65e1d457f0a39a914c410765f" exitCode=0
Sep 09 03:11:29.114457 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.114104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" event={"ID":"79426f9f-2ce5-411d-85f5-3f74010b0ac6","Type":"ContainerDied","Data":"f40d16cd16a97a7456f4247bcd0b159a8f29f3e65e1d457f0a39a914c410765f"}
Sep 09 03:11:29.145228 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.145204    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:11:29.285781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.285748    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-trusted-ca\") pod \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") "
Sep 09 03:11:29.285998 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.285811    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") pod \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") "
Sep 09 03:11:29.285998 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.285852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-installation-pull-secrets\") pod \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") "
Sep 09 03:11:29.285998 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.285870    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-bound-sa-token\") pod \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") "
Sep 09 03:11:29.286160 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.286052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79426f9f-2ce5-411d-85f5-3f74010b0ac6-ca-trust-extracted\") pod \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") "
Sep 09 03:11:29.286160 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.286096    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-certificates\") pod \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") "
Sep 09 03:11:29.286160 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.286129    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-image-registry-private-configuration\") pod \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") "
Sep 09 03:11:29.286160 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.286154    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx6xl\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-kube-api-access-kx6xl\") pod \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\" (UID: \"79426f9f-2ce5-411d-85f5-3f74010b0ac6\") "
Sep 09 03:11:29.286370 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.286325    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "79426f9f-2ce5-411d-85f5-3f74010b0ac6" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:29.286484 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.286469    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-trusted-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:11:29.286572 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.286488    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "79426f9f-2ce5-411d-85f5-3f74010b0ac6" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:29.295315 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.295277    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "79426f9f-2ce5-411d-85f5-3f74010b0ac6" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:11:29.295315 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.295303    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "79426f9f-2ce5-411d-85f5-3f74010b0ac6" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:11:29.295460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.295305    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "79426f9f-2ce5-411d-85f5-3f74010b0ac6" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:29.295507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.295479    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-kube-api-access-kx6xl" (OuterVolumeSpecName: "kube-api-access-kx6xl") pod "79426f9f-2ce5-411d-85f5-3f74010b0ac6" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6"). InnerVolumeSpecName "kube-api-access-kx6xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:11:29.295507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.295481    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "79426f9f-2ce5-411d-85f5-3f74010b0ac6" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:29.298822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.298794    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79426f9f-2ce5-411d-85f5-3f74010b0ac6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "79426f9f-2ce5-411d-85f5-3f74010b0ac6" (UID: "79426f9f-2ce5-411d-85f5-3f74010b0ac6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:11:29.386904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.386865    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:11:29.386904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.386900    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-installation-pull-secrets\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:11:29.386904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.386912    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-bound-sa-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:11:29.387129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.386921    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79426f9f-2ce5-411d-85f5-3f74010b0ac6-ca-trust-extracted\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:11:29.387129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.386930    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79426f9f-2ce5-411d-85f5-3f74010b0ac6-registry-certificates\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:11:29.387129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.386941    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79426f9f-2ce5-411d-85f5-3f74010b0ac6-image-registry-private-configuration\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:11:29.387129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:29.386950    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx6xl\" (UniqueName: \"kubernetes.io/projected/79426f9f-2ce5-411d-85f5-3f74010b0ac6-kube-api-access-kx6xl\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:11:30.120127 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:30.120086    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w" event={"ID":"79426f9f-2ce5-411d-85f5-3f74010b0ac6","Type":"ContainerDied","Data":"349eda69d8400c09934c02be47dd9a3e2f8165264d3e8e5f50d50a2940e12f30"}
Sep 09 03:11:30.120127 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:30.120115    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56749b4d77-d7p5w"
Sep 09 03:11:30.120664 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:30.120157    2562 scope.go:117] "RemoveContainer" containerID="f40d16cd16a97a7456f4247bcd0b159a8f29f3e65e1d457f0a39a914c410765f"
Sep 09 03:11:30.136035 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:30.136006    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56749b4d77-d7p5w"]
Sep 09 03:11:30.139616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:30.139585    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56749b4d77-d7p5w"]
Sep 09 03:11:30.371143 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:30.371101    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:30.428842 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:30.428803    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:31.786934 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:31.786895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:31.884362 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:31.884320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:31.936700 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:31.936664    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79426f9f-2ce5-411d-85f5-3f74010b0ac6" path="/var/lib/kubelet/pods/79426f9f-2ce5-411d-85f5-3f74010b0ac6/volumes"
Sep 09 03:11:37.518962 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:37.518923    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:37.519418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:37.518988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:37.523970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:37.523940    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:38.155733 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:38.155705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:11:38.199800 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:38.199759    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-698c548864-9gfcv"]
Sep 09 03:11:38.310024 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:38.309981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:38.411580 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:38.411469    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:38.511374 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:38.511340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:43.415663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:43.415567    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:43.466207 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:43.466174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:46.787787 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:46.787746    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:11:46.888624 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:11:46.888588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:03.220949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.220879    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-698c548864-9gfcv" podUID="e89852b3-e46d-4a48-8d0c-8a00823ee79c" containerName="console" containerID="cri-o://b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed" gracePeriod=15
Sep 09 03:12:03.470872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.470843    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:12:03.563866 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.563831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-oauth-config\") pod \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") "
Sep 09 03:12:03.564063 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.563879    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss6pl\" (UniqueName: \"kubernetes.io/projected/e89852b3-e46d-4a48-8d0c-8a00823ee79c-kube-api-access-ss6pl\") pod \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") "
Sep 09 03:12:03.564063 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.563909    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-service-ca\") pod \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") "
Sep 09 03:12:03.564063 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.563947    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-serving-cert\") pod \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") "
Sep 09 03:12:03.564063 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.563965    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-oauth-serving-cert\") pod \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") "
Sep 09 03:12:03.564063 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.564025    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-config\") pod \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") "
Sep 09 03:12:03.564063 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.564052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-trusted-ca-bundle\") pod \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\" (UID: \"e89852b3-e46d-4a48-8d0c-8a00823ee79c\") "
Sep 09 03:12:03.564468 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.564419    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e89852b3-e46d-4a48-8d0c-8a00823ee79c" (UID: "e89852b3-e46d-4a48-8d0c-8a00823ee79c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:12:03.564468 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.564427    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-service-ca" (OuterVolumeSpecName: "service-ca") pod "e89852b3-e46d-4a48-8d0c-8a00823ee79c" (UID: "e89852b3-e46d-4a48-8d0c-8a00823ee79c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:12:03.564859 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.564473    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-config" (OuterVolumeSpecName: "console-config") pod "e89852b3-e46d-4a48-8d0c-8a00823ee79c" (UID: "e89852b3-e46d-4a48-8d0c-8a00823ee79c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:12:03.564859 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.564595    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e89852b3-e46d-4a48-8d0c-8a00823ee79c" (UID: "e89852b3-e46d-4a48-8d0c-8a00823ee79c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:12:03.566278 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.566249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e89852b3-e46d-4a48-8d0c-8a00823ee79c" (UID: "e89852b3-e46d-4a48-8d0c-8a00823ee79c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:12:03.566399 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.566364    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e89852b3-e46d-4a48-8d0c-8a00823ee79c" (UID: "e89852b3-e46d-4a48-8d0c-8a00823ee79c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:12:03.566399 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.566385    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89852b3-e46d-4a48-8d0c-8a00823ee79c-kube-api-access-ss6pl" (OuterVolumeSpecName: "kube-api-access-ss6pl") pod "e89852b3-e46d-4a48-8d0c-8a00823ee79c" (UID: "e89852b3-e46d-4a48-8d0c-8a00823ee79c"). InnerVolumeSpecName "kube-api-access-ss6pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:12:03.665125 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.665077    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-config\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:03.665125 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.665111    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-trusted-ca-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:03.665125 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.665123    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-oauth-config\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:03.665125 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.665132    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss6pl\" (UniqueName: \"kubernetes.io/projected/e89852b3-e46d-4a48-8d0c-8a00823ee79c-kube-api-access-ss6pl\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:03.665388 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.665142    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:03.665388 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.665150    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e89852b3-e46d-4a48-8d0c-8a00823ee79c-console-serving-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:03.665388 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:03.665158    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e89852b3-e46d-4a48-8d0c-8a00823ee79c-oauth-serving-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:04.255351 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.255257    2562 generic.go:356] "Generic (PLEG): container finished" podID="e89852b3-e46d-4a48-8d0c-8a00823ee79c" containerID="b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed" exitCode=0
Sep 09 03:12:04.255351 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.255298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698c548864-9gfcv" event={"ID":"e89852b3-e46d-4a48-8d0c-8a00823ee79c","Type":"ContainerDied","Data":"b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed"}
Sep 09 03:12:04.255351 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.255308    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698c548864-9gfcv"
Sep 09 03:12:04.255351 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.255335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698c548864-9gfcv" event={"ID":"e89852b3-e46d-4a48-8d0c-8a00823ee79c","Type":"ContainerDied","Data":"44623fe99f7918482018d09e8ab6a21c2b61ac6c05bb4de7a78bb51f92d62c4a"}
Sep 09 03:12:04.255351 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.255354    2562 scope.go:117] "RemoveContainer" containerID="b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed"
Sep 09 03:12:04.263889 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.263869    2562 scope.go:117] "RemoveContainer" containerID="b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed"
Sep 09 03:12:04.264158 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:12:04.264138    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed\": container with ID starting with b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed not found: ID does not exist" containerID="b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed"
Sep 09 03:12:04.264204 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.264166    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed"} err="failed to get container status \"b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed\": rpc error: code = NotFound desc = could not find container \"b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed\": container with ID starting with b7f58b1076e7ac4151f174ae7093f4273a8ec179f3c51d2ff50f7f78e775a0ed not found: ID does not exist"
Sep 09 03:12:04.269848 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.269790    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-698c548864-9gfcv"]
Sep 09 03:12:04.271687 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:04.271663    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-698c548864-9gfcv"]
Sep 09 03:12:05.937260 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:05.937215    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89852b3-e46d-4a48-8d0c-8a00823ee79c" path="/var/lib/kubelet/pods/e89852b3-e46d-4a48-8d0c-8a00823ee79c/volumes"
Sep 09 03:12:23.962261 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:23.962219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:24.011719 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:24.011686    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:29.931590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:29.931509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:30.034930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.034895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:30.644164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.644128    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"]
Sep 09 03:12:30.644421 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.644409    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79426f9f-2ce5-411d-85f5-3f74010b0ac6" containerName="registry"
Sep 09 03:12:30.644465 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.644423    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79426f9f-2ce5-411d-85f5-3f74010b0ac6" containerName="registry"
Sep 09 03:12:30.644465 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.644434    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89852b3-e46d-4a48-8d0c-8a00823ee79c" containerName="console"
Sep 09 03:12:30.644465 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.644439    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89852b3-e46d-4a48-8d0c-8a00823ee79c" containerName="console"
Sep 09 03:12:30.644577 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.644484    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="79426f9f-2ce5-411d-85f5-3f74010b0ac6" containerName="registry"
Sep 09 03:12:30.644577 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.644494    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89852b3-e46d-4a48-8d0c-8a00823ee79c" containerName="console"
Sep 09 03:12:30.647517 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.647502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.650088 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.650067    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 03:12:30.650599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.650581    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9s8t2\""
Sep 09 03:12:30.650704 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.650602    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 03:12:30.656814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.656792    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"]
Sep 09 03:12:30.781899 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.781858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smc44\" (UniqueName: \"kubernetes.io/projected/833996ad-ea49-470f-986b-ff981e1f58b4-kube-api-access-smc44\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.781899 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.781905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.782133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.781943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.882668 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.882618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smc44\" (UniqueName: \"kubernetes.io/projected/833996ad-ea49-470f-986b-ff981e1f58b4-kube-api-access-smc44\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.882668 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.882671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.882829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.882711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.883152 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.883129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.883188 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.883142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.890738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.890703    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smc44\" (UniqueName: \"kubernetes.io/projected/833996ad-ea49-470f-986b-ff981e1f58b4-kube-api-access-smc44\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:30.956562 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:30.956433    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:31.230056 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:31.230031    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"]
Sep 09 03:12:31.232704 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:12:31.232666    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833996ad_ea49_470f_986b_ff981e1f58b4.slice/crio-acab9217ff6f164244bcc44b003b2696bc8e43caed7785b7af503c795c5c43b8 WatchSource:0}: Error finding container acab9217ff6f164244bcc44b003b2696bc8e43caed7785b7af503c795c5c43b8: Status 404 returned error can't find the container with id acab9217ff6f164244bcc44b003b2696bc8e43caed7785b7af503c795c5c43b8
Sep 09 03:12:31.234712 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:31.234696    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:12:31.365013 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:31.364976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x" event={"ID":"833996ad-ea49-470f-986b-ff981e1f58b4","Type":"ContainerStarted","Data":"acab9217ff6f164244bcc44b003b2696bc8e43caed7785b7af503c795c5c43b8"}
Sep 09 03:12:37.391603 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:37.391561    2562 generic.go:356] "Generic (PLEG): container finished" podID="833996ad-ea49-470f-986b-ff981e1f58b4" containerID="24b581f49eb67a48939595e9f1fe8696ee463f1c7d20db6b51a7c452bb6ba49b" exitCode=0
Sep 09 03:12:37.392018 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:37.391609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x" event={"ID":"833996ad-ea49-470f-986b-ff981e1f58b4","Type":"ContainerDied","Data":"24b581f49eb67a48939595e9f1fe8696ee463f1c7d20db6b51a7c452bb6ba49b"}
Sep 09 03:12:40.406686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:40.406648    2562 generic.go:356] "Generic (PLEG): container finished" podID="833996ad-ea49-470f-986b-ff981e1f58b4" containerID="a50795822bf5d8813f88cb9ca48532ede00b32720c169819a9c2e270dd5df3a3" exitCode=0
Sep 09 03:12:40.407120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:40.406737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x" event={"ID":"833996ad-ea49-470f-986b-ff981e1f58b4","Type":"ContainerDied","Data":"a50795822bf5d8813f88cb9ca48532ede00b32720c169819a9c2e270dd5df3a3"}
Sep 09 03:12:43.204344 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:43.204294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:43.260707 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:43.260668    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:47.438663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:47.438573    2562 generic.go:356] "Generic (PLEG): container finished" podID="833996ad-ea49-470f-986b-ff981e1f58b4" containerID="86aa464b4866bfc5213adfe414569e4889224b679ebce3b5a9624b7a65d0b5d6" exitCode=0
Sep 09 03:12:47.438663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:47.438645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x" event={"ID":"833996ad-ea49-470f-986b-ff981e1f58b4","Type":"ContainerDied","Data":"86aa464b4866bfc5213adfe414569e4889224b679ebce3b5a9624b7a65d0b5d6"}
Sep 09 03:12:48.571427 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.571399    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:48.645363 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.645330    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smc44\" (UniqueName: \"kubernetes.io/projected/833996ad-ea49-470f-986b-ff981e1f58b4-kube-api-access-smc44\") pod \"833996ad-ea49-470f-986b-ff981e1f58b4\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") "
Sep 09 03:12:48.645573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.645378    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-util\") pod \"833996ad-ea49-470f-986b-ff981e1f58b4\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") "
Sep 09 03:12:48.645573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.645445    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-bundle\") pod \"833996ad-ea49-470f-986b-ff981e1f58b4\" (UID: \"833996ad-ea49-470f-986b-ff981e1f58b4\") "
Sep 09 03:12:48.646319 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.646277    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-bundle" (OuterVolumeSpecName: "bundle") pod "833996ad-ea49-470f-986b-ff981e1f58b4" (UID: "833996ad-ea49-470f-986b-ff981e1f58b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:12:48.647803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.647773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833996ad-ea49-470f-986b-ff981e1f58b4-kube-api-access-smc44" (OuterVolumeSpecName: "kube-api-access-smc44") pod "833996ad-ea49-470f-986b-ff981e1f58b4" (UID: "833996ad-ea49-470f-986b-ff981e1f58b4"). InnerVolumeSpecName "kube-api-access-smc44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:12:48.650050 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.650022    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-util" (OuterVolumeSpecName: "util") pod "833996ad-ea49-470f-986b-ff981e1f58b4" (UID: "833996ad-ea49-470f-986b-ff981e1f58b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:12:48.746451 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.746352    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smc44\" (UniqueName: \"kubernetes.io/projected/833996ad-ea49-470f-986b-ff981e1f58b4-kube-api-access-smc44\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:48.746451 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.746383    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-util\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:48.746451 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:48.746393    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/833996ad-ea49-470f-986b-ff981e1f58b4-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:12:49.448381 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:49.448347    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x"
Sep 09 03:12:49.448590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:49.448349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cv69x" event={"ID":"833996ad-ea49-470f-986b-ff981e1f58b4","Type":"ContainerDied","Data":"acab9217ff6f164244bcc44b003b2696bc8e43caed7785b7af503c795c5c43b8"}
Sep 09 03:12:49.448590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:49.448466    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acab9217ff6f164244bcc44b003b2696bc8e43caed7785b7af503c795c5c43b8"
Sep 09 03:12:51.836646 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:51.836605    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:53.210929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.210894    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:53.803681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.803645    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"]
Sep 09 03:12:53.803950 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.803937    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833996ad-ea49-470f-986b-ff981e1f58b4" containerName="util"
Sep 09 03:12:53.803950 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.803951    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="833996ad-ea49-470f-986b-ff981e1f58b4" containerName="util"
Sep 09 03:12:53.804032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.803959    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833996ad-ea49-470f-986b-ff981e1f58b4" containerName="pull"
Sep 09 03:12:53.804032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.803965    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="833996ad-ea49-470f-986b-ff981e1f58b4" containerName="pull"
Sep 09 03:12:53.804032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.803971    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833996ad-ea49-470f-986b-ff981e1f58b4" containerName="extract"
Sep 09 03:12:53.804032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.803988    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="833996ad-ea49-470f-986b-ff981e1f58b4" containerName="extract"
Sep 09 03:12:53.804163 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.804036    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="833996ad-ea49-470f-986b-ff981e1f58b4" containerName="extract"
Sep 09 03:12:53.852846 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.852807    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"]
Sep 09 03:12:53.853011 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.852919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"
Sep 09 03:12:53.855446 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.855424    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6cckz\""
Sep 09 03:12:53.855582 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.855424    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:12:53.855763 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.855746    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 03:12:53.885674 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.885638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7vc\" (UniqueName: \"kubernetes.io/projected/41cfdddc-fe80-435a-9ff6-29f3683d5049-kube-api-access-xs7vc\") pod \"cert-manager-operator-controller-manager-55c444bf86-n9lhf\" (UID: \"41cfdddc-fe80-435a-9ff6-29f3683d5049\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"
Sep 09 03:12:53.885854 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.885681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41cfdddc-fe80-435a-9ff6-29f3683d5049-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n9lhf\" (UID: \"41cfdddc-fe80-435a-9ff6-29f3683d5049\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"
Sep 09 03:12:53.986844 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.986801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7vc\" (UniqueName: \"kubernetes.io/projected/41cfdddc-fe80-435a-9ff6-29f3683d5049-kube-api-access-xs7vc\") pod \"cert-manager-operator-controller-manager-55c444bf86-n9lhf\" (UID: \"41cfdddc-fe80-435a-9ff6-29f3683d5049\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"
Sep 09 03:12:53.986844 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.986849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41cfdddc-fe80-435a-9ff6-29f3683d5049-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n9lhf\" (UID: \"41cfdddc-fe80-435a-9ff6-29f3683d5049\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"
Sep 09 03:12:53.987250 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:53.987229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41cfdddc-fe80-435a-9ff6-29f3683d5049-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n9lhf\" (UID: \"41cfdddc-fe80-435a-9ff6-29f3683d5049\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"
Sep 09 03:12:54.000412 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:54.000385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7vc\" (UniqueName: \"kubernetes.io/projected/41cfdddc-fe80-435a-9ff6-29f3683d5049-kube-api-access-xs7vc\") pod \"cert-manager-operator-controller-manager-55c444bf86-n9lhf\" (UID: \"41cfdddc-fe80-435a-9ff6-29f3683d5049\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"
Sep 09 03:12:54.161829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:54.161732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"
Sep 09 03:12:54.459350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:54.459259    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf"]
Sep 09 03:12:54.462658 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:12:54.462630    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cfdddc_fe80_435a_9ff6_29f3683d5049.slice/crio-fa36822f70cf0277d1a45f8eae5aa203a076cbb081a53f12c711bf5060a6cac0 WatchSource:0}: Error finding container fa36822f70cf0277d1a45f8eae5aa203a076cbb081a53f12c711bf5060a6cac0: Status 404 returned error can't find the container with id fa36822f70cf0277d1a45f8eae5aa203a076cbb081a53f12c711bf5060a6cac0
Sep 09 03:12:54.469112 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:54.469081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf" event={"ID":"41cfdddc-fe80-435a-9ff6-29f3683d5049","Type":"ContainerStarted","Data":"fa36822f70cf0277d1a45f8eae5aa203a076cbb081a53f12c711bf5060a6cac0"}
Sep 09 03:12:55.442134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:55.442087    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:55.541986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:55.541929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:57.451456 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:57.451410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:57.483513 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:57.483469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf" event={"ID":"41cfdddc-fe80-435a-9ff6-29f3683d5049","Type":"ContainerStarted","Data":"2eb0bf991a75fa7659a765c841e32300f93a0fed823655e9dfc74e13ff86bbab"}
Sep 09 03:12:57.506070 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:57.506000    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n9lhf" podStartSLOduration=2.250136522 podStartE2EDuration="4.505979489s" totalImagesPullingTime="2.255842967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:12:57.504334069 +0000 UTC m=+426.066660615" watchObservedRunningTime="2026-09-09 03:12:57.505979489 +0000 UTC m=+426.068306058"
Sep 09 03:12:57.551443 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:57.551400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:57.892885 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:57.892845    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:58.050264 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.050227    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:58.483177 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.483140    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"]
Sep 09 03:12:58.509271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.509237    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"]
Sep 09 03:12:58.509439 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.509339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.511986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.511959    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 03:12:58.512106 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.511966    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9s8t2\""
Sep 09 03:12:58.512582 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.512564    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 03:12:58.625783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.625738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.625999 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.625810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gt7\" (UniqueName: \"kubernetes.io/projected/7f86ffde-a927-480d-a0fe-ee1656ce9705-kube-api-access-28gt7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.625999 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.625870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.726758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.726711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gt7\" (UniqueName: \"kubernetes.io/projected/7f86ffde-a927-480d-a0fe-ee1656ce9705-kube-api-access-28gt7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.726954 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.726787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.726954 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.726854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.727202 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.727181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.727241 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.727229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.734791 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.734712    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gt7\" (UniqueName: \"kubernetes.io/projected/7f86ffde-a927-480d-a0fe-ee1656ce9705-kube-api-access-28gt7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:58.818640 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:58.818596    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:12:59.111029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:59.110995    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"]
Sep 09 03:12:59.114011 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:12:59.113976    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f86ffde_a927_480d_a0fe_ee1656ce9705.slice/crio-494e71a11d704fca033f60c489e21afc931cc89cec10c1a1f7232c2c61728714 WatchSource:0}: Error finding container 494e71a11d704fca033f60c489e21afc931cc89cec10c1a1f7232c2c61728714: Status 404 returned error can't find the container with id 494e71a11d704fca033f60c489e21afc931cc89cec10c1a1f7232c2c61728714
Sep 09 03:12:59.450861 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:59.450746    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:12:59.494420 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:59.494380    2562 generic.go:356] "Generic (PLEG): container finished" podID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerID="6c6da5d053f81bf5dc2c8191181e8f1408231aeb3a45afb5c4d9d458d4d660ae" exitCode=0
Sep 09 03:12:59.494834 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:59.494468    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9" event={"ID":"7f86ffde-a927-480d-a0fe-ee1656ce9705","Type":"ContainerDied","Data":"6c6da5d053f81bf5dc2c8191181e8f1408231aeb3a45afb5c4d9d458d4d660ae"}
Sep 09 03:12:59.494834 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:59.494525    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9" event={"ID":"7f86ffde-a927-480d-a0fe-ee1656ce9705","Type":"ContainerStarted","Data":"494e71a11d704fca033f60c489e21afc931cc89cec10c1a1f7232c2c61728714"}
Sep 09 03:12:59.845264 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:12:59.845229    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:02.481724 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:02.481686    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:02.509860 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:02.509824    2562 generic.go:356] "Generic (PLEG): container finished" podID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerID="7c4efc52659b80e546826227c50223fe3dc1374ef887e9472576586eff68ba50" exitCode=0
Sep 09 03:13:02.510033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:02.509912    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9" event={"ID":"7f86ffde-a927-480d-a0fe-ee1656ce9705","Type":"ContainerDied","Data":"7c4efc52659b80e546826227c50223fe3dc1374ef887e9472576586eff68ba50"}
Sep 09 03:13:02.553252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:02.553202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:03.516377 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:03.516342    2562 generic.go:356] "Generic (PLEG): container finished" podID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerID="13a64c4225391e1e93f2c4cca614df2e602842eadd5bf974f7edfc37533cc8ea" exitCode=0
Sep 09 03:13:03.516834 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:03.516428    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9" event={"ID":"7f86ffde-a927-480d-a0fe-ee1656ce9705","Type":"ContainerDied","Data":"13a64c4225391e1e93f2c4cca614df2e602842eadd5bf974f7edfc37533cc8ea"}
Sep 09 03:13:04.227303 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.227268    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"]
Sep 09 03:13:04.229313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.229294    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"
Sep 09 03:13:04.231431 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.231409    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 03:13:04.232060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.232040    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wpqvb\""
Sep 09 03:13:04.232160 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.232050    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 03:13:04.242975 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.242951    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"]
Sep 09 03:13:04.275686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.275651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bc5\" (UniqueName: \"kubernetes.io/projected/8472d240-b526-411e-b153-37b88ce7533c-kube-api-access-67bc5\") pod \"cert-manager-cainjector-559cdd8c87-5qbff\" (UID: \"8472d240-b526-411e-b153-37b88ce7533c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"
Sep 09 03:13:04.275825 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.275713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8472d240-b526-411e-b153-37b88ce7533c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5qbff\" (UID: \"8472d240-b526-411e-b153-37b88ce7533c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"
Sep 09 03:13:04.376481 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.376444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bc5\" (UniqueName: \"kubernetes.io/projected/8472d240-b526-411e-b153-37b88ce7533c-kube-api-access-67bc5\") pod \"cert-manager-cainjector-559cdd8c87-5qbff\" (UID: \"8472d240-b526-411e-b153-37b88ce7533c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"
Sep 09 03:13:04.376666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.376508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8472d240-b526-411e-b153-37b88ce7533c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5qbff\" (UID: \"8472d240-b526-411e-b153-37b88ce7533c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"
Sep 09 03:13:04.384875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.384840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8472d240-b526-411e-b153-37b88ce7533c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5qbff\" (UID: \"8472d240-b526-411e-b153-37b88ce7533c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"
Sep 09 03:13:04.385039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.385019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bc5\" (UniqueName: \"kubernetes.io/projected/8472d240-b526-411e-b153-37b88ce7533c-kube-api-access-67bc5\") pod \"cert-manager-cainjector-559cdd8c87-5qbff\" (UID: \"8472d240-b526-411e-b153-37b88ce7533c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"
Sep 09 03:13:04.554224 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.554188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"
Sep 09 03:13:04.656872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.656846    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:13:04.781792 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.781693    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-bundle\") pod \"7f86ffde-a927-480d-a0fe-ee1656ce9705\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") "
Sep 09 03:13:04.781792 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.781781    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28gt7\" (UniqueName: \"kubernetes.io/projected/7f86ffde-a927-480d-a0fe-ee1656ce9705-kube-api-access-28gt7\") pod \"7f86ffde-a927-480d-a0fe-ee1656ce9705\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") "
Sep 09 03:13:04.782022 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.781814    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-util\") pod \"7f86ffde-a927-480d-a0fe-ee1656ce9705\" (UID: \"7f86ffde-a927-480d-a0fe-ee1656ce9705\") "
Sep 09 03:13:04.782196 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.782154    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-bundle" (OuterVolumeSpecName: "bundle") pod "7f86ffde-a927-480d-a0fe-ee1656ce9705" (UID: "7f86ffde-a927-480d-a0fe-ee1656ce9705"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:13:04.784111 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.784078    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f86ffde-a927-480d-a0fe-ee1656ce9705-kube-api-access-28gt7" (OuterVolumeSpecName: "kube-api-access-28gt7") pod "7f86ffde-a927-480d-a0fe-ee1656ce9705" (UID: "7f86ffde-a927-480d-a0fe-ee1656ce9705"). InnerVolumeSpecName "kube-api-access-28gt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:13:04.786445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.786423    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-util" (OuterVolumeSpecName: "util") pod "7f86ffde-a927-480d-a0fe-ee1656ce9705" (UID: "7f86ffde-a927-480d-a0fe-ee1656ce9705"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:13:04.831785 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.831762    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5qbff"]
Sep 09 03:13:04.834173 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:13:04.834141    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8472d240_b526_411e_b153_37b88ce7533c.slice/crio-645b748bd86b5db0281151aff8a34b484664e4b7c2c20b58ec6faf1f2286e8f9 WatchSource:0}: Error finding container 645b748bd86b5db0281151aff8a34b484664e4b7c2c20b58ec6faf1f2286e8f9: Status 404 returned error can't find the container with id 645b748bd86b5db0281151aff8a34b484664e4b7c2c20b58ec6faf1f2286e8f9
Sep 09 03:13:04.882945 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.882899    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:04.882945 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.882938    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28gt7\" (UniqueName: \"kubernetes.io/projected/7f86ffde-a927-480d-a0fe-ee1656ce9705-kube-api-access-28gt7\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:04.882945 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:04.882949    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f86ffde-a927-480d-a0fe-ee1656ce9705-util\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:05.527525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:05.527495    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9"
Sep 09 03:13:05.527726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:05.527498    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9m4d9" event={"ID":"7f86ffde-a927-480d-a0fe-ee1656ce9705","Type":"ContainerDied","Data":"494e71a11d704fca033f60c489e21afc931cc89cec10c1a1f7232c2c61728714"}
Sep 09 03:13:05.527726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:05.527595    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="494e71a11d704fca033f60c489e21afc931cc89cec10c1a1f7232c2c61728714"
Sep 09 03:13:05.529078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:05.529048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff" event={"ID":"8472d240-b526-411e-b153-37b88ce7533c","Type":"ContainerStarted","Data":"645b748bd86b5db0281151aff8a34b484664e4b7c2c20b58ec6faf1f2286e8f9"}
Sep 09 03:13:07.541336 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:07.541299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff" event={"ID":"8472d240-b526-411e-b153-37b88ce7533c","Type":"ContainerStarted","Data":"4b5b0deacc5e9008360000b46b774259d8016bafaee091bc143f302864337ea8"}
Sep 09 03:13:07.556813 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:07.556766    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5qbff" podStartSLOduration=1.024386108 podStartE2EDuration="3.556749665s" totalImagesPullingTime="2.532363557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:07.556019932 +0000 UTC m=+436.118346477" watchObservedRunningTime="2026-09-09 03:13:07.556749665 +0000 UTC m=+436.119076210"
Sep 09 03:13:08.128957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:08.128925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:08.229248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:08.229208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:08.329849 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:08.329808    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:10.322729 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:10.322690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:10.426476 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:10.426434    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:16.413508 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:16.413418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:16.513227 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:16.513183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:18.082349 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.082316    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"]
Sep 09 03:13:18.082823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.082673    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerName="util"
Sep 09 03:13:18.082823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.082689    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerName="util"
Sep 09 03:13:18.082823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.082705    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerName="pull"
Sep 09 03:13:18.082823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.082710    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerName="pull"
Sep 09 03:13:18.082823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.082716    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerName="extract"
Sep 09 03:13:18.082823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.082722    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerName="extract"
Sep 09 03:13:18.082823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.082778    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f86ffde-a927-480d-a0fe-ee1656ce9705" containerName="extract"
Sep 09 03:13:18.085840 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.085820    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.088296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.088278    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 03:13:18.088898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.088876    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 03:13:18.089039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.088883    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9s8t2\""
Sep 09 03:13:18.093828 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.093804    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"]
Sep 09 03:13:18.188662 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.188634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vs4\" (UniqueName: \"kubernetes.io/projected/8afab067-2e2c-4ebf-a478-ab63a04dd044-kube-api-access-76vs4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.188662 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.188668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.188855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.188717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.270658 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.270623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:18.289189 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.289163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76vs4\" (UniqueName: \"kubernetes.io/projected/8afab067-2e2c-4ebf-a478-ab63a04dd044-kube-api-access-76vs4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.289299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.289195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.289400 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.289379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.289567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.289529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.289676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.289660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.296659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.296634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vs4\" (UniqueName: \"kubernetes.io/projected/8afab067-2e2c-4ebf-a478-ab63a04dd044-kube-api-access-76vs4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.396748 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.396665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:18.410783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.410754    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mpxrs"]
Sep 09 03:13:18.414685 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.414665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mpxrs"
Sep 09 03:13:18.416822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.416804    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wbkx7\""
Sep 09 03:13:18.422258 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.422234    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mpxrs"]
Sep 09 03:13:18.491744 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.491702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88sjc\" (UniqueName: \"kubernetes.io/projected/d52b0593-520c-4e6f-8800-c586ea06c54d-kube-api-access-88sjc\") pod \"cert-manager-55b4d954d5-mpxrs\" (UID: \"d52b0593-520c-4e6f-8800-c586ea06c54d\") " pod="cert-manager/cert-manager-55b4d954d5-mpxrs"
Sep 09 03:13:18.491909 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.491799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d52b0593-520c-4e6f-8800-c586ea06c54d-bound-sa-token\") pod \"cert-manager-55b4d954d5-mpxrs\" (UID: \"d52b0593-520c-4e6f-8800-c586ea06c54d\") " pod="cert-manager/cert-manager-55b4d954d5-mpxrs"
Sep 09 03:13:18.592844 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.592815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d52b0593-520c-4e6f-8800-c586ea06c54d-bound-sa-token\") pod \"cert-manager-55b4d954d5-mpxrs\" (UID: \"d52b0593-520c-4e6f-8800-c586ea06c54d\") " pod="cert-manager/cert-manager-55b4d954d5-mpxrs"
Sep 09 03:13:18.592984 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.592866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88sjc\" (UniqueName: \"kubernetes.io/projected/d52b0593-520c-4e6f-8800-c586ea06c54d-kube-api-access-88sjc\") pod \"cert-manager-55b4d954d5-mpxrs\" (UID: \"d52b0593-520c-4e6f-8800-c586ea06c54d\") " pod="cert-manager/cert-manager-55b4d954d5-mpxrs"
Sep 09 03:13:18.605037 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.605009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d52b0593-520c-4e6f-8800-c586ea06c54d-bound-sa-token\") pod \"cert-manager-55b4d954d5-mpxrs\" (UID: \"d52b0593-520c-4e6f-8800-c586ea06c54d\") " pod="cert-manager/cert-manager-55b4d954d5-mpxrs"
Sep 09 03:13:18.605182 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.605105    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88sjc\" (UniqueName: \"kubernetes.io/projected/d52b0593-520c-4e6f-8800-c586ea06c54d-kube-api-access-88sjc\") pod \"cert-manager-55b4d954d5-mpxrs\" (UID: \"d52b0593-520c-4e6f-8800-c586ea06c54d\") " pod="cert-manager/cert-manager-55b4d954d5-mpxrs"
Sep 09 03:13:18.670722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.670659    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"]
Sep 09 03:13:18.673303 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:13:18.673279    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afab067_2e2c_4ebf_a478_ab63a04dd044.slice/crio-e8160ae1d197b5be22daa512ef4159367a66ce0634e0d3c245750e6f596dd1c8 WatchSource:0}: Error finding container e8160ae1d197b5be22daa512ef4159367a66ce0634e0d3c245750e6f596dd1c8: Status 404 returned error can't find the container with id e8160ae1d197b5be22daa512ef4159367a66ce0634e0d3c245750e6f596dd1c8
Sep 09 03:13:18.725772 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.725747    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mpxrs"
Sep 09 03:13:18.998260 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:18.998173    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mpxrs"]
Sep 09 03:13:19.001248 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:13:19.001222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd52b0593_520c_4e6f_8800_c586ea06c54d.slice/crio-2112b3f01e1c1eb0a319a676e8984dcb34b1817e246fd05f3bfba405f87afed7 WatchSource:0}: Error finding container 2112b3f01e1c1eb0a319a676e8984dcb34b1817e246fd05f3bfba405f87afed7: Status 404 returned error can't find the container with id 2112b3f01e1c1eb0a319a676e8984dcb34b1817e246fd05f3bfba405f87afed7
Sep 09 03:13:19.592947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:19.592910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mpxrs" event={"ID":"d52b0593-520c-4e6f-8800-c586ea06c54d","Type":"ContainerStarted","Data":"e6f7b9f1a7ca8926315aa4e67acfbe84194799c872c85602705a1310edfc3cd3"}
Sep 09 03:13:19.593360 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:19.592958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mpxrs" event={"ID":"d52b0593-520c-4e6f-8800-c586ea06c54d","Type":"ContainerStarted","Data":"2112b3f01e1c1eb0a319a676e8984dcb34b1817e246fd05f3bfba405f87afed7"}
Sep 09 03:13:19.594593 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:19.594564    2562 generic.go:356] "Generic (PLEG): container finished" podID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerID="4ac67bc3dbca580b93a0382d5b7d57b4f73f8e0cd91a21584ac765db366a5bad" exitCode=0
Sep 09 03:13:19.594700 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:19.594634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5" event={"ID":"8afab067-2e2c-4ebf-a478-ab63a04dd044","Type":"ContainerDied","Data":"4ac67bc3dbca580b93a0382d5b7d57b4f73f8e0cd91a21584ac765db366a5bad"}
Sep 09 03:13:19.594700 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:19.594677    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5" event={"ID":"8afab067-2e2c-4ebf-a478-ab63a04dd044","Type":"ContainerStarted","Data":"e8160ae1d197b5be22daa512ef4159367a66ce0634e0d3c245750e6f596dd1c8"}
Sep 09 03:13:19.606697 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:19.606650    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mpxrs" podStartSLOduration=1.606633 podStartE2EDuration="1.606633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:19.606198559 +0000 UTC m=+448.168525106" watchObservedRunningTime="2026-09-09 03:13:19.606633 +0000 UTC m=+448.168959557"
Sep 09 03:13:20.601070 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:20.601038    2562 generic.go:356] "Generic (PLEG): container finished" podID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerID="476854ca11feb1842384f3a2175423b3369605a0ebba5c6a53fb7f92ea380334" exitCode=0
Sep 09 03:13:20.601441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:20.601127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5" event={"ID":"8afab067-2e2c-4ebf-a478-ab63a04dd044","Type":"ContainerDied","Data":"476854ca11feb1842384f3a2175423b3369605a0ebba5c6a53fb7f92ea380334"}
Sep 09 03:13:21.607942 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:21.607901    2562 generic.go:356] "Generic (PLEG): container finished" podID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerID="a938dfa99e16f6a84c0b6fbe4bbd5dce14790b9115495b1d7267e17ddbe99576" exitCode=0
Sep 09 03:13:21.608359 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:21.607964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5" event={"ID":"8afab067-2e2c-4ebf-a478-ab63a04dd044","Type":"ContainerDied","Data":"a938dfa99e16f6a84c0b6fbe4bbd5dce14790b9115495b1d7267e17ddbe99576"}
Sep 09 03:13:22.740405 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.740378    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:22.830788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.830750    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76vs4\" (UniqueName: \"kubernetes.io/projected/8afab067-2e2c-4ebf-a478-ab63a04dd044-kube-api-access-76vs4\") pod \"8afab067-2e2c-4ebf-a478-ab63a04dd044\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") "
Sep 09 03:13:22.830962 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.830826    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-util\") pod \"8afab067-2e2c-4ebf-a478-ab63a04dd044\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") "
Sep 09 03:13:22.830962 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.830866    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-bundle\") pod \"8afab067-2e2c-4ebf-a478-ab63a04dd044\" (UID: \"8afab067-2e2c-4ebf-a478-ab63a04dd044\") "
Sep 09 03:13:22.831768 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.831733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-bundle" (OuterVolumeSpecName: "bundle") pod "8afab067-2e2c-4ebf-a478-ab63a04dd044" (UID: "8afab067-2e2c-4ebf-a478-ab63a04dd044"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:13:22.833136 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.833099    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afab067-2e2c-4ebf-a478-ab63a04dd044-kube-api-access-76vs4" (OuterVolumeSpecName: "kube-api-access-76vs4") pod "8afab067-2e2c-4ebf-a478-ab63a04dd044" (UID: "8afab067-2e2c-4ebf-a478-ab63a04dd044"). InnerVolumeSpecName "kube-api-access-76vs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:13:22.836882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.836860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-util" (OuterVolumeSpecName: "util") pod "8afab067-2e2c-4ebf-a478-ab63a04dd044" (UID: "8afab067-2e2c-4ebf-a478-ab63a04dd044"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:13:22.931734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.931644    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76vs4\" (UniqueName: \"kubernetes.io/projected/8afab067-2e2c-4ebf-a478-ab63a04dd044-kube-api-access-76vs4\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:22.931734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.931675    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-util\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:22.931734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:22.931686    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8afab067-2e2c-4ebf-a478-ab63a04dd044-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:23.619199 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:23.619161    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5"
Sep 09 03:13:23.619348 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:23.619169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dns5" event={"ID":"8afab067-2e2c-4ebf-a478-ab63a04dd044","Type":"ContainerDied","Data":"e8160ae1d197b5be22daa512ef4159367a66ce0634e0d3c245750e6f596dd1c8"}
Sep 09 03:13:23.619348 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:23.619277    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8160ae1d197b5be22daa512ef4159367a66ce0634e0d3c245750e6f596dd1c8"
Sep 09 03:13:24.152824 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:24.152792    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:24.253381 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:24.253343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:34.280439 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:34.280395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:34.371226 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:34.371181    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:34.903266 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:34.903232    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:35.180259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.180154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:35.276512 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.276474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:35.307638 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.307603    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"]
Sep 09 03:13:35.308133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.307903    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerName="pull"
Sep 09 03:13:35.308133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.307914    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerName="pull"
Sep 09 03:13:35.308133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.307926    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerName="util"
Sep 09 03:13:35.308133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.307932    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerName="util"
Sep 09 03:13:35.308133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.307944    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerName="extract"
Sep 09 03:13:35.308133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.307950    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerName="extract"
Sep 09 03:13:35.308133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.308002    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afab067-2e2c-4ebf-a478-ab63a04dd044" containerName="extract"
Sep 09 03:13:35.315895 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.315862    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"]
Sep 09 03:13:35.316074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.316010    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.318293 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.318270    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 03:13:35.318825 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.318804    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 03:13:35.318941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.318852    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 03:13:35.319061 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.319044    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7r62m\""
Sep 09 03:13:35.319105 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.319094    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 03:13:35.319353 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.319340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.321340 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.321323    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9s8t2\""
Sep 09 03:13:35.321418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.321339    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 03:13:35.321418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.321386    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 03:13:35.330673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.330651    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"]
Sep 09 03:13:35.332516 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.332492    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"]
Sep 09 03:13:35.431964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.431862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.431964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.431905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.431964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.431930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grs89\" (UniqueName: \"kubernetes.io/projected/e1265e1e-8e1f-4049-8008-3ecd0023c364-kube-api-access-grs89\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.431964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.431948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.432257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.432018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsh8m\" (UniqueName: \"kubernetes.io/projected/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-kube-api-access-rsh8m\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.432257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.432063    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.432257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.432101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.432257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.432175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.503328 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.503284    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:35.533338 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.533338 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.533636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.533636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grs89\" (UniqueName: \"kubernetes.io/projected/e1265e1e-8e1f-4049-8008-3ecd0023c364-kube-api-access-grs89\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.533636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.533636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsh8m\" (UniqueName: \"kubernetes.io/projected/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-kube-api-access-rsh8m\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.533636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.533636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.533958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.533958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.533958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.533939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.535994 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.535963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.536231 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.536214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.536273 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.536239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.542250 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.542223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsh8m\" (UniqueName: \"kubernetes.io/projected/fe4b17cd-7c22-4e8c-8310-980a7ca13c85-kube-api-access-rsh8m\") pod \"opendatahub-operator-controller-manager-856874fcb4-75hs5\" (UID: \"fe4b17cd-7c22-4e8c-8310-980a7ca13c85\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.542739 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.542721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grs89\" (UniqueName: \"kubernetes.io/projected/e1265e1e-8e1f-4049-8008-3ecd0023c364-kube-api-access-grs89\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.582884 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.582837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:35.627776 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.627741    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:35.634251 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.634222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:35.683186 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.683085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:35.785245 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.785205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:35.885710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.885669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:35.910959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.910924    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"]
Sep 09 03:13:35.915765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:35.915739    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"]
Sep 09 03:13:35.916906 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:13:35.916881    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4b17cd_7c22_4e8c_8310_980a7ca13c85.slice/crio-92e31bb4bc9265474b5c1fd1cca39accfaa2bb2032db8e63c8b96b96410822ed WatchSource:0}: Error finding container 92e31bb4bc9265474b5c1fd1cca39accfaa2bb2032db8e63c8b96b96410822ed: Status 404 returned error can't find the container with id 92e31bb4bc9265474b5c1fd1cca39accfaa2bb2032db8e63c8b96b96410822ed
Sep 09 03:13:35.917552 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:13:35.917518    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1265e1e_8e1f_4049_8008_3ecd0023c364.slice/crio-6c7bd3fa2aee51cb631df21730591010b1bb2065a1d5e91c70cc99f0ef91b027 WatchSource:0}: Error finding container 6c7bd3fa2aee51cb631df21730591010b1bb2065a1d5e91c70cc99f0ef91b027: Status 404 returned error can't find the container with id 6c7bd3fa2aee51cb631df21730591010b1bb2065a1d5e91c70cc99f0ef91b027
Sep 09 03:13:36.680881 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:36.680840    2562 generic.go:356] "Generic (PLEG): container finished" podID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerID="e93d350ded2b0a7716d449a4fc863a4d625eed2122d2f126a10c7a048558499f" exitCode=0
Sep 09 03:13:36.681601 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:36.680928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj" event={"ID":"e1265e1e-8e1f-4049-8008-3ecd0023c364","Type":"ContainerDied","Data":"e93d350ded2b0a7716d449a4fc863a4d625eed2122d2f126a10c7a048558499f"}
Sep 09 03:13:36.681601 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:36.680988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj" event={"ID":"e1265e1e-8e1f-4049-8008-3ecd0023c364","Type":"ContainerStarted","Data":"6c7bd3fa2aee51cb631df21730591010b1bb2065a1d5e91c70cc99f0ef91b027"}
Sep 09 03:13:36.682998 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:36.682935    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5" event={"ID":"fe4b17cd-7c22-4e8c-8310-980a7ca13c85","Type":"ContainerStarted","Data":"92e31bb4bc9265474b5c1fd1cca39accfaa2bb2032db8e63c8b96b96410822ed"}
Sep 09 03:13:37.691456 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:37.691414    2562 generic.go:356] "Generic (PLEG): container finished" podID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerID="a4116ec86a6b385ea26482846e281ef05a256d0b219effd6c00c5dcbb4e625f4" exitCode=0
Sep 09 03:13:37.691911 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:37.691494    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj" event={"ID":"e1265e1e-8e1f-4049-8008-3ecd0023c364","Type":"ContainerDied","Data":"a4116ec86a6b385ea26482846e281ef05a256d0b219effd6c00c5dcbb4e625f4"}
Sep 09 03:13:39.702482 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:39.702440    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5" event={"ID":"fe4b17cd-7c22-4e8c-8310-980a7ca13c85","Type":"ContainerStarted","Data":"35f959684947952f0f2cc6d06c2bff1a898f7317b0edf4d9db6510af9237c24b"}
Sep 09 03:13:39.704726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:39.704696    2562 generic.go:356] "Generic (PLEG): container finished" podID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerID="b166be5ac9154610e14f16d29c31bb2265008efb3ada98110f64a8747fa5053c" exitCode=0
Sep 09 03:13:39.704852 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:39.704777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj" event={"ID":"e1265e1e-8e1f-4049-8008-3ecd0023c364","Type":"ContainerDied","Data":"b166be5ac9154610e14f16d29c31bb2265008efb3ada98110f64a8747fa5053c"}
Sep 09 03:13:40.711661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.711566    2562 generic.go:356] "Generic (PLEG): container finished" podID="fe4b17cd-7c22-4e8c-8310-980a7ca13c85" containerID="35f959684947952f0f2cc6d06c2bff1a898f7317b0edf4d9db6510af9237c24b" exitCode=0
Sep 09 03:13:40.711661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.711649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5" event={"ID":"fe4b17cd-7c22-4e8c-8310-980a7ca13c85","Type":"ContainerDied","Data":"35f959684947952f0f2cc6d06c2bff1a898f7317b0edf4d9db6510af9237c24b"}
Sep 09 03:13:40.837703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.837674    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:40.979313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.979207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-bundle\") pod \"e1265e1e-8e1f-4049-8008-3ecd0023c364\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") "
Sep 09 03:13:40.979313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.979292    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grs89\" (UniqueName: \"kubernetes.io/projected/e1265e1e-8e1f-4049-8008-3ecd0023c364-kube-api-access-grs89\") pod \"e1265e1e-8e1f-4049-8008-3ecd0023c364\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") "
Sep 09 03:13:40.979592 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.979331    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-util\") pod \"e1265e1e-8e1f-4049-8008-3ecd0023c364\" (UID: \"e1265e1e-8e1f-4049-8008-3ecd0023c364\") "
Sep 09 03:13:40.980090 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.980059    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-bundle" (OuterVolumeSpecName: "bundle") pod "e1265e1e-8e1f-4049-8008-3ecd0023c364" (UID: "e1265e1e-8e1f-4049-8008-3ecd0023c364"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:13:40.981757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.981734    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1265e1e-8e1f-4049-8008-3ecd0023c364-kube-api-access-grs89" (OuterVolumeSpecName: "kube-api-access-grs89") pod "e1265e1e-8e1f-4049-8008-3ecd0023c364" (UID: "e1265e1e-8e1f-4049-8008-3ecd0023c364"). InnerVolumeSpecName "kube-api-access-grs89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:13:40.986341 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:40.986313    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-util" (OuterVolumeSpecName: "util") pod "e1265e1e-8e1f-4049-8008-3ecd0023c364" (UID: "e1265e1e-8e1f-4049-8008-3ecd0023c364"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:13:41.080093 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.080066    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-util\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:41.080093 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.080090    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1265e1e-8e1f-4049-8008-3ecd0023c364-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:41.080299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.080099    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grs89\" (UniqueName: \"kubernetes.io/projected/e1265e1e-8e1f-4049-8008-3ecd0023c364-kube-api-access-grs89\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:13:41.718416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.718385    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj"
Sep 09 03:13:41.718853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.718419    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05l2fj" event={"ID":"e1265e1e-8e1f-4049-8008-3ecd0023c364","Type":"ContainerDied","Data":"6c7bd3fa2aee51cb631df21730591010b1bb2065a1d5e91c70cc99f0ef91b027"}
Sep 09 03:13:41.718853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.718469    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c7bd3fa2aee51cb631df21730591010b1bb2065a1d5e91c70cc99f0ef91b027"
Sep 09 03:13:41.720690 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.720661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5" event={"ID":"fe4b17cd-7c22-4e8c-8310-980a7ca13c85","Type":"ContainerStarted","Data":"de9cf3411932d87ca9be0a4b4f4d4be6fe24fcb9d0ab9e69600f97a153b62641"}
Sep 09 03:13:41.720831 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.720799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:41.740312 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:41.740254    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5" podStartSLOduration=2.085486818 podStartE2EDuration="6.740234027s" totalImagesPullingTime="3.529074871s" totalInitContainerRuntime="1.125672338s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:41.738715704 +0000 UTC m=+470.301042248" watchObservedRunningTime="2026-09-09 03:13:41.740234027 +0000 UTC m=+470.302560574"
Sep 09 03:13:47.819658 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:47.819617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:47.902852 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:47.902810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:52.728417 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:52.728386    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-75hs5"
Sep 09 03:13:54.236075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.236033    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:54.358782 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.358745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:54.563730 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.563686    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:54.597886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.597855    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"]
Sep 09 03:13:54.598156 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.598144    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerName="extract"
Sep 09 03:13:54.598211 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.598158    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerName="extract"
Sep 09 03:13:54.598211 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.598183    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerName="util"
Sep 09 03:13:54.598211 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.598189    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerName="util"
Sep 09 03:13:54.598211 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.598195    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerName="pull"
Sep 09 03:13:54.598211 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.598200    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerName="pull"
Sep 09 03:13:54.598354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.598244    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1265e1e-8e1f-4049-8008-3ecd0023c364" containerName="extract"
Sep 09 03:13:54.606682 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.606659    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.612275 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.612227    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 03:13:54.612655 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.612627    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 03:13:54.633253 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.633219    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"]
Sep 09 03:13:54.692524 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.692491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ac1f072-6090-4fd1-bb05-27a571187828-tmp\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.692720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.692531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pccz\" (UniqueName: \"kubernetes.io/projected/9ac1f072-6090-4fd1-bb05-27a571187828-kube-api-access-9pccz\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.692720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.692582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac1f072-6090-4fd1-bb05-27a571187828-tls-certs\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.793646 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.793611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pccz\" (UniqueName: \"kubernetes.io/projected/9ac1f072-6090-4fd1-bb05-27a571187828-kube-api-access-9pccz\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.793842 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.793655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac1f072-6090-4fd1-bb05-27a571187828-tls-certs\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.793842 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.793822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ac1f072-6090-4fd1-bb05-27a571187828-tmp\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.796189 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.796160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ac1f072-6090-4fd1-bb05-27a571187828-tmp\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.796391 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.796375    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac1f072-6090-4fd1-bb05-27a571187828-tls-certs\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.810365 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.810338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pccz\" (UniqueName: \"kubernetes.io/projected/9ac1f072-6090-4fd1-bb05-27a571187828-kube-api-access-9pccz\") pod \"kube-auth-proxy-5c8d5f99fb-74wrc\" (UID: \"9ac1f072-6090-4fd1-bb05-27a571187828\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:54.817965 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.817896    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:54.916446 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:54.916407    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"
Sep 09 03:13:55.190754 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:55.190729    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc"]
Sep 09 03:13:55.193438 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:13:55.193409    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac1f072_6090_4fd1_bb05_27a571187828.slice/crio-cdb6b6860750afab90d66bab7e6e8dc3976e0c0da7f2e09a92ffebdc359dae0c WatchSource:0}: Error finding container cdb6b6860750afab90d66bab7e6e8dc3976e0c0da7f2e09a92ffebdc359dae0c: Status 404 returned error can't find the container with id cdb6b6860750afab90d66bab7e6e8dc3976e0c0da7f2e09a92ffebdc359dae0c
Sep 09 03:13:55.788268 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:55.788231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc" event={"ID":"9ac1f072-6090-4fd1-bb05-27a571187828","Type":"ContainerStarted","Data":"cdb6b6860750afab90d66bab7e6e8dc3976e0c0da7f2e09a92ffebdc359dae0c"}
Sep 09 03:13:56.085846 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.085721    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"]
Sep 09 03:13:56.087590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.087527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:56.089661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.089636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.092460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.091788    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 03:13:56.092460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.091849    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 03:13:56.092460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.091854    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 03:13:56.092460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.091872    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 03:13:56.092460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.092277    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c6p6n\""
Sep 09 03:13:56.105947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.104653    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.105947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.104704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.105947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.104764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gg5c\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-kube-api-access-9gg5c\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.105947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.104792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.105947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.104826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.105947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.104850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.105947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.104890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2894871c-31cb-4df8-9e87-ee63b59ce2f7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.107005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.106975    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"]
Sep 09 03:13:56.157631 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.157495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:56.206349 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.206308    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.206581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.206360    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.206581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.206411    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2894871c-31cb-4df8-9e87-ee63b59ce2f7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.206581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.206480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.206581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.206518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.206806 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.206603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gg5c\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-kube-api-access-9gg5c\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.206806 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.206628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.207625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.207455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.209918 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.209868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.210823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.210210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2894871c-31cb-4df8-9e87-ee63b59ce2f7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.210823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.210490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.210823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.210771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.228555 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.228493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.228861 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.228807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gg5c\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-kube-api-access-9gg5c\") pod \"istiod-openshift-gateway-7b7995d5cd-2pv6k\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.406193 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.406063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:13:56.700632 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:56.700420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"]
Sep 09 03:13:57.013246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:57.013109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:57.520893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:57.520844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:57.981666 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:13:57.981633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2894871c_31cb_4df8_9e87_ee63b59ce2f7.slice/crio-1736f070e982bb75c41c301fc09336eff89d533910dbd02431761ab20064990f WatchSource:0}: Error finding container 1736f070e982bb75c41c301fc09336eff89d533910dbd02431761ab20064990f: Status 404 returned error can't find the container with id 1736f070e982bb75c41c301fc09336eff89d533910dbd02431761ab20064990f
Sep 09 03:13:58.805875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:58.805839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc" event={"ID":"9ac1f072-6090-4fd1-bb05-27a571187828","Type":"ContainerStarted","Data":"2dd3c44f0b89778a5a58273ff6b6c98017b5a78e956f7c57a677246958316258"}
Sep 09 03:13:58.807120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:58.807094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k" event={"ID":"2894871c-31cb-4df8-9e87-ee63b59ce2f7","Type":"ContainerStarted","Data":"1736f070e982bb75c41c301fc09336eff89d533910dbd02431761ab20064990f"}
Sep 09 03:13:58.822584 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:13:58.822514    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-74wrc" podStartSLOduration=1.975830998 podStartE2EDuration="4.822498261s" totalImagesPullingTime="2.846667263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:58.820367612 +0000 UTC m=+487.382694157" watchObservedRunningTime="2026-09-09 03:13:58.822498261 +0000 UTC m=+487.384824806"
Sep 09 03:14:00.791617 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:00.791561    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:14:00.792052 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:00.791650    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:14:00.976183 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:00.976134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:01.077688 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:01.077582    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:01.711358 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:01.711308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:01.811445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:01.811414    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:01.824945 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:01.824908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k" event={"ID":"2894871c-31cb-4df8-9e87-ee63b59ce2f7","Type":"ContainerStarted","Data":"52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490"}
Sep 09 03:14:01.825151 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:01.825124    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:14:01.826785 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:01.826762    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2pv6k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:14:01.826883 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:01.826809    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k" podUID="2894871c-31cb-4df8-9e87-ee63b59ce2f7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:14:01.848873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:01.848817    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k" podStartSLOduration=3.041492283 podStartE2EDuration="5.848798275s" totalImagesPullingTime="2.807305992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:01.847739053 +0000 UTC m=+490.410065830" watchObservedRunningTime="2026-09-09 03:14:01.848798275 +0000 UTC m=+490.411124819"
Sep 09 03:14:02.288565 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:02.288247    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:02.477680 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:02.477643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:02.651428 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:02.651340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:02.824673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:02.824634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:02.830254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:02.830231    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:14:02.937645 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:02.937554    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:05.162418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:05.162374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:05.251808 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:05.251748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:05.504207 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:05.504105    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:05.634388 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:05.634348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:05.733732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:05.733684    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:14.755882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.755844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:14.834413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.834378    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-759nz"]
Sep 09 03:14:14.836759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.836736    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:14.839901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.839880    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hzxrx\""
Sep 09 03:14:14.840259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.840244    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 09 03:14:14.860415 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.860381    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-759nz"]
Sep 09 03:14:14.875836 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.875776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:14.882928 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.882897    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"]
Sep 09 03:14:14.885365 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.885341    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:14.892217 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.892188    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 03:14:14.898410 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.898384    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rvw7m\""
Sep 09 03:14:14.909270 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.909231    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"]
Sep 09 03:14:14.980983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.976969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsw4v\" (UniqueName: \"kubernetes.io/projected/d4f3a691-d396-4ace-8db8-bc69767f5bb4-kube-api-access-dsw4v\") pod \"kserve-controller-manager-d6f4b65b8-759nz\" (UID: \"d4f3a691-d396-4ace-8db8-bc69767f5bb4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:14.980983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.977032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4f3a691-d396-4ace-8db8-bc69767f5bb4-cert\") pod \"kserve-controller-manager-d6f4b65b8-759nz\" (UID: \"d4f3a691-d396-4ace-8db8-bc69767f5bb4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:14.980983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.977064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de70f2e3-d26d-47b1-9f70-d0a1daf67b25-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x5vf8\" (UID: \"de70f2e3-d26d-47b1-9f70-d0a1daf67b25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:14.980983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.977125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdp4r\" (UniqueName: \"kubernetes.io/projected/de70f2e3-d26d-47b1-9f70-d0a1daf67b25-kube-api-access-cdp4r\") pod \"llmisvc-controller-manager-777c6bd7d5-x5vf8\" (UID: \"de70f2e3-d26d-47b1-9f70-d0a1daf67b25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:14.995663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:14.995626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:15.078008 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.077970    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsw4v\" (UniqueName: \"kubernetes.io/projected/d4f3a691-d396-4ace-8db8-bc69767f5bb4-kube-api-access-dsw4v\") pod \"kserve-controller-manager-d6f4b65b8-759nz\" (UID: \"d4f3a691-d396-4ace-8db8-bc69767f5bb4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:15.078213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.078015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4f3a691-d396-4ace-8db8-bc69767f5bb4-cert\") pod \"kserve-controller-manager-d6f4b65b8-759nz\" (UID: \"d4f3a691-d396-4ace-8db8-bc69767f5bb4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:15.078213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.078044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de70f2e3-d26d-47b1-9f70-d0a1daf67b25-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x5vf8\" (UID: \"de70f2e3-d26d-47b1-9f70-d0a1daf67b25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:15.078424 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.078392    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdp4r\" (UniqueName: \"kubernetes.io/projected/de70f2e3-d26d-47b1-9f70-d0a1daf67b25-kube-api-access-cdp4r\") pod \"llmisvc-controller-manager-777c6bd7d5-x5vf8\" (UID: \"de70f2e3-d26d-47b1-9f70-d0a1daf67b25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:15.080513 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.080481    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:15.081160 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.081138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de70f2e3-d26d-47b1-9f70-d0a1daf67b25-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x5vf8\" (UID: \"de70f2e3-d26d-47b1-9f70-d0a1daf67b25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:15.081339 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.081318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4f3a691-d396-4ace-8db8-bc69767f5bb4-cert\") pod \"kserve-controller-manager-d6f4b65b8-759nz\" (UID: \"d4f3a691-d396-4ace-8db8-bc69767f5bb4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:15.090423 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.090388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsw4v\" (UniqueName: \"kubernetes.io/projected/d4f3a691-d396-4ace-8db8-bc69767f5bb4-kube-api-access-dsw4v\") pod \"kserve-controller-manager-d6f4b65b8-759nz\" (UID: \"d4f3a691-d396-4ace-8db8-bc69767f5bb4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:15.092277 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.092248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdp4r\" (UniqueName: \"kubernetes.io/projected/de70f2e3-d26d-47b1-9f70-d0a1daf67b25-kube-api-access-cdp4r\") pod \"llmisvc-controller-manager-777c6bd7d5-x5vf8\" (UID: \"de70f2e3-d26d-47b1-9f70-d0a1daf67b25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:15.147439 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.147390    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:15.150101 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.150078    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:15.196091 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.196045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:15.251811 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.251777    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:15.434784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.434698    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-759nz"]
Sep 09 03:14:15.437655 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:14:15.437621    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f3a691_d396_4ace_8db8_bc69767f5bb4.slice/crio-aa9da3851cd1907293637b49f0aee9bce55144fcef0b48d6235a5eaf221bf69d WatchSource:0}: Error finding container aa9da3851cd1907293637b49f0aee9bce55144fcef0b48d6235a5eaf221bf69d: Status 404 returned error can't find the container with id aa9da3851cd1907293637b49f0aee9bce55144fcef0b48d6235a5eaf221bf69d
Sep 09 03:14:15.480590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.480557    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"]
Sep 09 03:14:15.483172 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:14:15.483141    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podde70f2e3_d26d_47b1_9f70_d0a1daf67b25.slice/crio-26362a64942cbdf945815927d747365c3cfdd2889b85f0bfdead37e39d55ee1d WatchSource:0}: Error finding container 26362a64942cbdf945815927d747365c3cfdd2889b85f0bfdead37e39d55ee1d: Status 404 returned error can't find the container with id 26362a64942cbdf945815927d747365c3cfdd2889b85f0bfdead37e39d55ee1d
Sep 09 03:14:15.899569 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.899512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz" event={"ID":"d4f3a691-d396-4ace-8db8-bc69767f5bb4","Type":"ContainerStarted","Data":"aa9da3851cd1907293637b49f0aee9bce55144fcef0b48d6235a5eaf221bf69d"}
Sep 09 03:14:15.900637 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:15.900613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8" event={"ID":"de70f2e3-d26d-47b1-9f70-d0a1daf67b25","Type":"ContainerStarted","Data":"26362a64942cbdf945815927d747365c3cfdd2889b85f0bfdead37e39d55ee1d"}
Sep 09 03:14:16.931146 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:16.931049    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"]
Sep 09 03:14:16.931830 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:16.931336    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k" podUID="2894871c-31cb-4df8-9e87-ee63b59ce2f7" containerName="discovery" containerID="cri-o://52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490" gracePeriod=30
Sep 09 03:14:17.658105 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.658068    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:14:17.803229 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.803186    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2894871c-31cb-4df8-9e87-ee63b59ce2f7-local-certs\") pod \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") "
Sep 09 03:14:17.803421 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.803239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-token\") pod \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") "
Sep 09 03:14:17.803421 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.803321    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-kubeconfig\") pod \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") "
Sep 09 03:14:17.803421 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.803360    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-cacerts\") pod \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") "
Sep 09 03:14:17.803421 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.803400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-ca-configmap\") pod \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") "
Sep 09 03:14:17.803676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.803446    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-dns-cert\") pod \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") "
Sep 09 03:14:17.803676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.803481    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gg5c\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-kube-api-access-9gg5c\") pod \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\" (UID: \"2894871c-31cb-4df8-9e87-ee63b59ce2f7\") "
Sep 09 03:14:17.805237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.804884    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "2894871c-31cb-4df8-9e87-ee63b59ce2f7" (UID: "2894871c-31cb-4df8-9e87-ee63b59ce2f7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:14:17.807114 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.807025    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "2894871c-31cb-4df8-9e87-ee63b59ce2f7" (UID: "2894871c-31cb-4df8-9e87-ee63b59ce2f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:14:17.810587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.808021    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-cacerts" (OuterVolumeSpecName: "cacerts") pod "2894871c-31cb-4df8-9e87-ee63b59ce2f7" (UID: "2894871c-31cb-4df8-9e87-ee63b59ce2f7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:14:17.810587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.808103    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "2894871c-31cb-4df8-9e87-ee63b59ce2f7" (UID: "2894871c-31cb-4df8-9e87-ee63b59ce2f7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:14:17.810587 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.809603    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2894871c-31cb-4df8-9e87-ee63b59ce2f7-local-certs" (OuterVolumeSpecName: "local-certs") pod "2894871c-31cb-4df8-9e87-ee63b59ce2f7" (UID: "2894871c-31cb-4df8-9e87-ee63b59ce2f7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:14:17.815585 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.815524    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-kube-api-access-9gg5c" (OuterVolumeSpecName: "kube-api-access-9gg5c") pod "2894871c-31cb-4df8-9e87-ee63b59ce2f7" (UID: "2894871c-31cb-4df8-9e87-ee63b59ce2f7"). InnerVolumeSpecName "kube-api-access-9gg5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:14:17.815585 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.815563    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "2894871c-31cb-4df8-9e87-ee63b59ce2f7" (UID: "2894871c-31cb-4df8-9e87-ee63b59ce2f7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:14:17.904827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.904741    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-kubeconfig\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:14:17.904827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.904771    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-cacerts\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:14:17.904827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.904781    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-ca-configmap\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:14:17.904827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.904790    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-csr-dns-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:14:17.904827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.904800    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gg5c\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-kube-api-access-9gg5c\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:14:17.904827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.904811    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2894871c-31cb-4df8-9e87-ee63b59ce2f7-local-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:14:17.904827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.904824    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2894871c-31cb-4df8-9e87-ee63b59ce2f7-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:14:17.916237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.914578    2562 generic.go:356] "Generic (PLEG): container finished" podID="2894871c-31cb-4df8-9e87-ee63b59ce2f7" containerID="52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490" exitCode=0
Sep 09 03:14:17.916237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.914636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k" event={"ID":"2894871c-31cb-4df8-9e87-ee63b59ce2f7","Type":"ContainerDied","Data":"52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490"}
Sep 09 03:14:17.916237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.914716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k" event={"ID":"2894871c-31cb-4df8-9e87-ee63b59ce2f7","Type":"ContainerDied","Data":"1736f070e982bb75c41c301fc09336eff89d533910dbd02431761ab20064990f"}
Sep 09 03:14:17.916237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.914716    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"
Sep 09 03:14:17.916237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.914731    2562 scope.go:117] "RemoveContainer" containerID="52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490"
Sep 09 03:14:17.940095 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.940056    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"]
Sep 09 03:14:17.946430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:17.946404    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2pv6k"]
Sep 09 03:14:18.258604 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:18.258386    2562 scope.go:117] "RemoveContainer" containerID="52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490"
Sep 09 03:14:18.258810 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:14:18.258789    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490\": container with ID starting with 52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490 not found: ID does not exist" containerID="52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490"
Sep 09 03:14:18.258872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:18.258820    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490"} err="failed to get container status \"52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490\": rpc error: code = NotFound desc = could not find container \"52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490\": container with ID starting with 52006bbf0491ab4a1c5068e08094c5ad730f5481156e48cedddf72e801552490 not found: ID does not exist"
Sep 09 03:14:18.922082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:18.922045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz" event={"ID":"d4f3a691-d396-4ace-8db8-bc69767f5bb4","Type":"ContainerStarted","Data":"3d2e3d1499c0595e139c17542a84e35c7971a74c9659e2b943444133b4a0be55"}
Sep 09 03:14:18.922290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:18.922118    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:14:18.923812 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:18.923763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8" event={"ID":"de70f2e3-d26d-47b1-9f70-d0a1daf67b25","Type":"ContainerStarted","Data":"3625e0c2284c7ae29d9b5c161eb43b0cffbcff1ca4d9bb5481237c8303a69bdf"}
Sep 09 03:14:18.923926 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:18.923835    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:18.938824 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:18.938773    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz" podStartSLOduration=2.111437571 podStartE2EDuration="4.938755924s" totalImagesPullingTime="2.827318353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:18.936868846 +0000 UTC m=+507.499195391" watchObservedRunningTime="2026-09-09 03:14:18.938755924 +0000 UTC m=+507.501082470"
Sep 09 03:14:18.951865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:18.951808    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8" podStartSLOduration=2.166856752 podStartE2EDuration="4.951786546s" totalImagesPullingTime="2.784929794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:18.951396946 +0000 UTC m=+507.513723545" watchObservedRunningTime="2026-09-09 03:14:18.951786546 +0000 UTC m=+507.514113092"
Sep 09 03:14:19.937465 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:19.937429    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2894871c-31cb-4df8-9e87-ee63b59ce2f7" path="/var/lib/kubelet/pods/2894871c-31cb-4df8-9e87-ee63b59ce2f7/volumes"
Sep 09 03:14:46.505074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:46.504990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:49.941526 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:49.941496    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x5vf8"
Sep 09 03:14:49.941983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:14:49.941578    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-759nz"
Sep 09 03:15:06.404015 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:06.403971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:07.846363 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:07.846325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:09.817625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.817592    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"]
Sep 09 03:15:09.818003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.817903    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2894871c-31cb-4df8-9e87-ee63b59ce2f7" containerName="discovery"
Sep 09 03:15:09.818003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.817914    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2894871c-31cb-4df8-9e87-ee63b59ce2f7" containerName="discovery"
Sep 09 03:15:09.818003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.817977    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2894871c-31cb-4df8-9e87-ee63b59ce2f7" containerName="discovery"
Sep 09 03:15:09.822953 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.822934    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:09.825309 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.825287    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 03:15:09.825447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.825340    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 03:15:09.825447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.825410    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9s8t2\""
Sep 09 03:15:09.828568 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.828524    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"]
Sep 09 03:15:09.915341 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.915310    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"]
Sep 09 03:15:09.917551 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.917525    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:09.925414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.925365    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"]
Sep 09 03:15:09.937166 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.937133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:09.937334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.937213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:09.937334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:09.937280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8gh\" (UniqueName: \"kubernetes.io/projected/b19253ac-5ba2-44b0-96d5-664a647032b5-kube-api-access-jg8gh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:10.012858 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.012826    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"]
Sep 09 03:15:10.015257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.015241    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.025113 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.025080    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"]
Sep 09 03:15:10.037931 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.037899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8gh\" (UniqueName: \"kubernetes.io/projected/b19253ac-5ba2-44b0-96d5-664a647032b5-kube-api-access-jg8gh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:10.038104 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.037957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.038104 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.038010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:10.038104 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.038047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:10.038104 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.038074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rvd\" (UniqueName: \"kubernetes.io/projected/db23328d-b4c2-4578-a7bf-9b2a11bb8038-kube-api-access-j8rvd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.038330 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.038239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.038432 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.038414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:10.038504 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.038482    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:10.045336 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.045309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8gh\" (UniqueName: \"kubernetes.io/projected/b19253ac-5ba2-44b0-96d5-664a647032b5-kube-api-access-jg8gh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:10.113415 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.113336    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"]
Sep 09 03:15:10.115856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.115837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.125892 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.125868    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"]
Sep 09 03:15:10.133056 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.133035    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:10.139376 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.139347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.139492 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.139408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.139492 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.139452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rvd\" (UniqueName: \"kubernetes.io/projected/db23328d-b4c2-4578-a7bf-9b2a11bb8038-kube-api-access-j8rvd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.139492 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.139475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxqtk\" (UniqueName: \"kubernetes.io/projected/9a03da44-5369-4691-9de4-b62bc44c7583-kube-api-access-dxqtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.139713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.139564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.139713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.139607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.139814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.139764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.139873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.139844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.147766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.147742    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rvd\" (UniqueName: \"kubernetes.io/projected/db23328d-b4c2-4578-a7bf-9b2a11bb8038-kube-api-access-j8rvd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.227017 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.226986    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:10.240719 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.240683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxqtk\" (UniqueName: \"kubernetes.io/projected/9a03da44-5369-4691-9de4-b62bc44c7583-kube-api-access-dxqtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.240848 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.240746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.240848 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.240817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58g8m\" (UniqueName: \"kubernetes.io/projected/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-kube-api-access-58g8m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.240970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.240863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.240970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.240895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.240970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.240965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.241167 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.241145    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.241335 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.241315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.249134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.249111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxqtk\" (UniqueName: \"kubernetes.io/projected/9a03da44-5369-4691-9de4-b62bc44c7583-kube-api-access-dxqtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.331785 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.331753    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:10.341983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.341913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58g8m\" (UniqueName: \"kubernetes.io/projected/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-kube-api-access-58g8m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.341983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.341965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.342099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.341992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.342342 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.342319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.342409 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.342362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.350319 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.350283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58g8m\" (UniqueName: \"kubernetes.io/projected/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-kube-api-access-58g8m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.407917 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.407893    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"]
Sep 09 03:15:10.409265 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:10.409236    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19253ac_5ba2_44b0_96d5_664a647032b5.slice/crio-f7437b455ea64c1caeb487bd826c1686edec0f6a8afe6495467f7e5f71dfd59d WatchSource:0}: Error finding container f7437b455ea64c1caeb487bd826c1686edec0f6a8afe6495467f7e5f71dfd59d: Status 404 returned error can't find the container with id f7437b455ea64c1caeb487bd826c1686edec0f6a8afe6495467f7e5f71dfd59d
Sep 09 03:15:10.426111 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.426090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:10.510185 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.510134    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"]
Sep 09 03:15:10.511150 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:10.511113    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb23328d_b4c2_4578_a7bf_9b2a11bb8038.slice/crio-105c83906ac7f71bc473bfde768ef98abe19586047459ccd649c390ea8f67763 WatchSource:0}: Error finding container 105c83906ac7f71bc473bfde768ef98abe19586047459ccd649c390ea8f67763: Status 404 returned error can't find the container with id 105c83906ac7f71bc473bfde768ef98abe19586047459ccd649c390ea8f67763
Sep 09 03:15:10.608517 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.608493    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"]
Sep 09 03:15:10.610267 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:10.610237    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a03da44_5369_4691_9de4_b62bc44c7583.slice/crio-7324e8cb9ade01b0d5172effd58031362fc310e64d9d83825e0c9660fb947658 WatchSource:0}: Error finding container 7324e8cb9ade01b0d5172effd58031362fc310e64d9d83825e0c9660fb947658: Status 404 returned error can't find the container with id 7324e8cb9ade01b0d5172effd58031362fc310e64d9d83825e0c9660fb947658
Sep 09 03:15:10.701895 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:10.701868    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"]
Sep 09 03:15:10.756417 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:10.756382    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b78ab0_55ce_4b85_b659_fd93a0b5ec7c.slice/crio-ca2bbb9cf33896c4ac71e66e340569728687ec7697293afbb72fb2f888c973e9 WatchSource:0}: Error finding container ca2bbb9cf33896c4ac71e66e340569728687ec7697293afbb72fb2f888c973e9: Status 404 returned error can't find the container with id ca2bbb9cf33896c4ac71e66e340569728687ec7697293afbb72fb2f888c973e9
Sep 09 03:15:11.173065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.173032    2562 generic.go:356] "Generic (PLEG): container finished" podID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerID="9f9cdae2c4ca872994c75c23d0c2e394b4ac3c12fd3093274a27e5320967b0ce" exitCode=0
Sep 09 03:15:11.173504 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.173170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f" event={"ID":"b19253ac-5ba2-44b0-96d5-664a647032b5","Type":"ContainerDied","Data":"9f9cdae2c4ca872994c75c23d0c2e394b4ac3c12fd3093274a27e5320967b0ce"}
Sep 09 03:15:11.173504 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.173231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f" event={"ID":"b19253ac-5ba2-44b0-96d5-664a647032b5","Type":"ContainerStarted","Data":"f7437b455ea64c1caeb487bd826c1686edec0f6a8afe6495467f7e5f71dfd59d"}
Sep 09 03:15:11.174827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.174795    2562 generic.go:356] "Generic (PLEG): container finished" podID="9a03da44-5369-4691-9de4-b62bc44c7583" containerID="29b7115601d4da70f76078381f849f020e2ba18a290adbf349464749663ca864" exitCode=0
Sep 09 03:15:11.174907 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.174840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9" event={"ID":"9a03da44-5369-4691-9de4-b62bc44c7583","Type":"ContainerDied","Data":"29b7115601d4da70f76078381f849f020e2ba18a290adbf349464749663ca864"}
Sep 09 03:15:11.174964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.174904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9" event={"ID":"9a03da44-5369-4691-9de4-b62bc44c7583","Type":"ContainerStarted","Data":"7324e8cb9ade01b0d5172effd58031362fc310e64d9d83825e0c9660fb947658"}
Sep 09 03:15:11.176613 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.176584    2562 generic.go:356] "Generic (PLEG): container finished" podID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerID="29a7619afd65d66f3795b353af866b24d6ac8a35c358f44709632117ff5f1c00" exitCode=0
Sep 09 03:15:11.176711 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.176679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh" event={"ID":"db23328d-b4c2-4578-a7bf-9b2a11bb8038","Type":"ContainerDied","Data":"29a7619afd65d66f3795b353af866b24d6ac8a35c358f44709632117ff5f1c00"}
Sep 09 03:15:11.176772 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.176726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh" event={"ID":"db23328d-b4c2-4578-a7bf-9b2a11bb8038","Type":"ContainerStarted","Data":"105c83906ac7f71bc473bfde768ef98abe19586047459ccd649c390ea8f67763"}
Sep 09 03:15:11.178504 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.178479    2562 generic.go:356] "Generic (PLEG): container finished" podID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerID="949c7ac30aa50446bcda38b6a45de410ffb64da7d973da267204a42ce6b5f28b" exitCode=0
Sep 09 03:15:11.178602 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.178514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj" event={"ID":"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c","Type":"ContainerDied","Data":"949c7ac30aa50446bcda38b6a45de410ffb64da7d973da267204a42ce6b5f28b"}
Sep 09 03:15:11.178602 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.178568    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj" event={"ID":"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c","Type":"ContainerStarted","Data":"ca2bbb9cf33896c4ac71e66e340569728687ec7697293afbb72fb2f888c973e9"}
Sep 09 03:15:11.555769 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.555737    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d9578b6c7-zmgw2"]
Sep 09 03:15:11.558090 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.558067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.570643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.570618    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d9578b6c7-zmgw2"]
Sep 09 03:15:11.601252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.601218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:11.652780 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.652740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-trusted-ca-bundle\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.652780 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.652785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/489fcd11-027c-445e-a133-1d61ad2dfca7-console-oauth-config\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.652966 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.652811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/489fcd11-027c-445e-a133-1d61ad2dfca7-console-serving-cert\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.652966 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.652832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktbb\" (UniqueName: \"kubernetes.io/projected/489fcd11-027c-445e-a133-1d61ad2dfca7-kube-api-access-xktbb\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.652966 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.652855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-service-ca\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.652966 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.652942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-oauth-serving-cert\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.653114 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.652979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-console-config\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.659588 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.659528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:11.754181 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.754142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/489fcd11-027c-445e-a133-1d61ad2dfca7-console-serving-cert\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.754181 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.754184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktbb\" (UniqueName: \"kubernetes.io/projected/489fcd11-027c-445e-a133-1d61ad2dfca7-kube-api-access-xktbb\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.754416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.754216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-service-ca\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.754416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.754269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-oauth-serving-cert\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.754416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.754305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-console-config\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.754416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.754338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-trusted-ca-bundle\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.754416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.754372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/489fcd11-027c-445e-a133-1d61ad2dfca7-console-oauth-config\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.755120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.755095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-console-config\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.755214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.755121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-service-ca\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.755214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.755121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-oauth-serving-cert\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.755525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.755503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489fcd11-027c-445e-a133-1d61ad2dfca7-trusted-ca-bundle\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.756941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.756920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/489fcd11-027c-445e-a133-1d61ad2dfca7-console-serving-cert\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.757128 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.757106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/489fcd11-027c-445e-a133-1d61ad2dfca7-console-oauth-config\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.762425 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.762402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktbb\" (UniqueName: \"kubernetes.io/projected/489fcd11-027c-445e-a133-1d61ad2dfca7-kube-api-access-xktbb\") pod \"console-7d9578b6c7-zmgw2\" (UID: \"489fcd11-027c-445e-a133-1d61ad2dfca7\") " pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:11.867863 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:11.867776    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:12.138612 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:12.138581    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d9578b6c7-zmgw2"]
Sep 09 03:15:12.187367 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:12.187337    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489fcd11_027c_445e_a133_1d61ad2dfca7.slice/crio-03056ff7c92408f157cf9533af459007ee94eee49ab62ee4256cf0c656ad4cb4 WatchSource:0}: Error finding container 03056ff7c92408f157cf9533af459007ee94eee49ab62ee4256cf0c656ad4cb4: Status 404 returned error can't find the container with id 03056ff7c92408f157cf9533af459007ee94eee49ab62ee4256cf0c656ad4cb4
Sep 09 03:15:13.191978 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.191940    2562 generic.go:356] "Generic (PLEG): container finished" podID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerID="54a1939386cd7368c8c5ef5723aca25c30a90653113fec62a466e32962fbc8fd" exitCode=0
Sep 09 03:15:13.192443 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.192019    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj" event={"ID":"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c","Type":"ContainerDied","Data":"54a1939386cd7368c8c5ef5723aca25c30a90653113fec62a466e32962fbc8fd"}
Sep 09 03:15:13.194286 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.194258    2562 generic.go:356] "Generic (PLEG): container finished" podID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerID="a73835389dca396340b71216c5ae5d35cb773bd43f476eb342e071e630c50759" exitCode=0
Sep 09 03:15:13.194381 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.194357    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f" event={"ID":"b19253ac-5ba2-44b0-96d5-664a647032b5","Type":"ContainerDied","Data":"a73835389dca396340b71216c5ae5d35cb773bd43f476eb342e071e630c50759"}
Sep 09 03:15:13.196618 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.196584    2562 generic.go:356] "Generic (PLEG): container finished" podID="9a03da44-5369-4691-9de4-b62bc44c7583" containerID="331b4a8becd332f57d3697742b38e1f0655fd589325035ee859784a3a2638f9d" exitCode=0
Sep 09 03:15:13.196706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.196635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9" event={"ID":"9a03da44-5369-4691-9de4-b62bc44c7583","Type":"ContainerDied","Data":"331b4a8becd332f57d3697742b38e1f0655fd589325035ee859784a3a2638f9d"}
Sep 09 03:15:13.198763 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.198744    2562 generic.go:356] "Generic (PLEG): container finished" podID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerID="8ac272997bdcd41652ae9fe167ffb1a420c2ca8257bbfc282091be3d8b6ab5cf" exitCode=0
Sep 09 03:15:13.198862 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.198820    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh" event={"ID":"db23328d-b4c2-4578-a7bf-9b2a11bb8038","Type":"ContainerDied","Data":"8ac272997bdcd41652ae9fe167ffb1a420c2ca8257bbfc282091be3d8b6ab5cf"}
Sep 09 03:15:13.200584 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.200563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9578b6c7-zmgw2" event={"ID":"489fcd11-027c-445e-a133-1d61ad2dfca7","Type":"ContainerStarted","Data":"e5e256980d0fcb65049ed437a898c18d89d9579f7ca8c7bfee56cc86de3d7e9c"}
Sep 09 03:15:13.200649 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.200597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9578b6c7-zmgw2" event={"ID":"489fcd11-027c-445e-a133-1d61ad2dfca7","Type":"ContainerStarted","Data":"03056ff7c92408f157cf9533af459007ee94eee49ab62ee4256cf0c656ad4cb4"}
Sep 09 03:15:13.237317 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:13.237272    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d9578b6c7-zmgw2" podStartSLOduration=2.23725757 podStartE2EDuration="2.23725757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:13.236416331 +0000 UTC m=+561.798742855" watchObservedRunningTime="2026-09-09 03:15:13.23725757 +0000 UTC m=+561.799584114"
Sep 09 03:15:14.207817 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:14.207786    2562 generic.go:356] "Generic (PLEG): container finished" podID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerID="f8af53e71e1fada41589f7b7c51e3d80806107cc80c695cc1532b7c071807c0e" exitCode=0
Sep 09 03:15:14.208150 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:14.207870    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f" event={"ID":"b19253ac-5ba2-44b0-96d5-664a647032b5","Type":"ContainerDied","Data":"f8af53e71e1fada41589f7b7c51e3d80806107cc80c695cc1532b7c071807c0e"}
Sep 09 03:15:14.210090 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:14.210068    2562 generic.go:356] "Generic (PLEG): container finished" podID="9a03da44-5369-4691-9de4-b62bc44c7583" containerID="16187c5759259270655a6c0886ea9bce2511441dbfcd0326fc7adb3c34f68488" exitCode=0
Sep 09 03:15:14.210202 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:14.210133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9" event={"ID":"9a03da44-5369-4691-9de4-b62bc44c7583","Type":"ContainerDied","Data":"16187c5759259270655a6c0886ea9bce2511441dbfcd0326fc7adb3c34f68488"}
Sep 09 03:15:14.212399 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:14.212381    2562 generic.go:356] "Generic (PLEG): container finished" podID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerID="18b57e3e45f5c5c5ddca824785d0c5a090152a7aa0bd4b90b5c07a43c2bbf8a5" exitCode=0
Sep 09 03:15:14.212476 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:14.212458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh" event={"ID":"db23328d-b4c2-4578-a7bf-9b2a11bb8038","Type":"ContainerDied","Data":"18b57e3e45f5c5c5ddca824785d0c5a090152a7aa0bd4b90b5c07a43c2bbf8a5"}
Sep 09 03:15:14.214625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:14.214604    2562 generic.go:356] "Generic (PLEG): container finished" podID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerID="7b62a24d03d45ec28bf9bfd7b4f4d4b4349bbe18d5a07139b5ff79ab6be22e97" exitCode=0
Sep 09 03:15:14.214705 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:14.214666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj" event={"ID":"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c","Type":"ContainerDied","Data":"7b62a24d03d45ec28bf9bfd7b4f4d4b4349bbe18d5a07139b5ff79ab6be22e97"}
Sep 09 03:15:15.376875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.376849    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:15.426083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.426057    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:15.428830 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.428810    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:15.432386 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.432367    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:15.490274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490200    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-util\") pod \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") "
Sep 09 03:15:15.490403 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxqtk\" (UniqueName: \"kubernetes.io/projected/9a03da44-5369-4691-9de4-b62bc44c7583-kube-api-access-dxqtk\") pod \"9a03da44-5369-4691-9de4-b62bc44c7583\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") "
Sep 09 03:15:15.490403 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490363    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-bundle\") pod \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") "
Sep 09 03:15:15.490403 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-bundle\") pod \"b19253ac-5ba2-44b0-96d5-664a647032b5\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") "
Sep 09 03:15:15.490565 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-util\") pod \"9a03da44-5369-4691-9de4-b62bc44c7583\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") "
Sep 09 03:15:15.490565 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-util\") pod \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") "
Sep 09 03:15:15.490565 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg8gh\" (UniqueName: \"kubernetes.io/projected/b19253ac-5ba2-44b0-96d5-664a647032b5-kube-api-access-jg8gh\") pod \"b19253ac-5ba2-44b0-96d5-664a647032b5\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") "
Sep 09 03:15:15.490565 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-util\") pod \"b19253ac-5ba2-44b0-96d5-664a647032b5\" (UID: \"b19253ac-5ba2-44b0-96d5-664a647032b5\") "
Sep 09 03:15:15.490565 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490528    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8rvd\" (UniqueName: \"kubernetes.io/projected/db23328d-b4c2-4578-a7bf-9b2a11bb8038-kube-api-access-j8rvd\") pod \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") "
Sep 09 03:15:15.490796 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490570    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-bundle\") pod \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\" (UID: \"db23328d-b4c2-4578-a7bf-9b2a11bb8038\") "
Sep 09 03:15:15.490796 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490593    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58g8m\" (UniqueName: \"kubernetes.io/projected/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-kube-api-access-58g8m\") pod \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\" (UID: \"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c\") "
Sep 09 03:15:15.490796 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.490642    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-bundle\") pod \"9a03da44-5369-4691-9de4-b62bc44c7583\" (UID: \"9a03da44-5369-4691-9de4-b62bc44c7583\") "
Sep 09 03:15:15.491706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.491205    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-bundle" (OuterVolumeSpecName: "bundle") pod "b19253ac-5ba2-44b0-96d5-664a647032b5" (UID: "b19253ac-5ba2-44b0-96d5-664a647032b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:15.491706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.491395    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-bundle" (OuterVolumeSpecName: "bundle") pod "93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" (UID: "93b78ab0-55ce-4b85-b659-fd93a0b5ec7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:15.491706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.491415    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-bundle" (OuterVolumeSpecName: "bundle") pod "db23328d-b4c2-4578-a7bf-9b2a11bb8038" (UID: "db23328d-b4c2-4578-a7bf-9b2a11bb8038"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:15.492001 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.491955    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-bundle" (OuterVolumeSpecName: "bundle") pod "9a03da44-5369-4691-9de4-b62bc44c7583" (UID: "9a03da44-5369-4691-9de4-b62bc44c7583"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:15.493681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.493650    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a03da44-5369-4691-9de4-b62bc44c7583-kube-api-access-dxqtk" (OuterVolumeSpecName: "kube-api-access-dxqtk") pod "9a03da44-5369-4691-9de4-b62bc44c7583" (UID: "9a03da44-5369-4691-9de4-b62bc44c7583"). InnerVolumeSpecName "kube-api-access-dxqtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:15:15.493799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.493774    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-kube-api-access-58g8m" (OuterVolumeSpecName: "kube-api-access-58g8m") pod "93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" (UID: "93b78ab0-55ce-4b85-b659-fd93a0b5ec7c"). InnerVolumeSpecName "kube-api-access-58g8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:15:15.494225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.494192    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db23328d-b4c2-4578-a7bf-9b2a11bb8038-kube-api-access-j8rvd" (OuterVolumeSpecName: "kube-api-access-j8rvd") pod "db23328d-b4c2-4578-a7bf-9b2a11bb8038" (UID: "db23328d-b4c2-4578-a7bf-9b2a11bb8038"). InnerVolumeSpecName "kube-api-access-j8rvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:15:15.494705 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.494687    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19253ac-5ba2-44b0-96d5-664a647032b5-kube-api-access-jg8gh" (OuterVolumeSpecName: "kube-api-access-jg8gh") pod "b19253ac-5ba2-44b0-96d5-664a647032b5" (UID: "b19253ac-5ba2-44b0-96d5-664a647032b5"). InnerVolumeSpecName "kube-api-access-jg8gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:15:15.497796 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.497762    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-util" (OuterVolumeSpecName: "util") pod "93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" (UID: "93b78ab0-55ce-4b85-b659-fd93a0b5ec7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:15.497928 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.497907    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-util" (OuterVolumeSpecName: "util") pod "db23328d-b4c2-4578-a7bf-9b2a11bb8038" (UID: "db23328d-b4c2-4578-a7bf-9b2a11bb8038"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:15.498106 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.498088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-util" (OuterVolumeSpecName: "util") pod "b19253ac-5ba2-44b0-96d5-664a647032b5" (UID: "b19253ac-5ba2-44b0-96d5-664a647032b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:15.498225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.498204    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-util" (OuterVolumeSpecName: "util") pod "9a03da44-5369-4691-9de4-b62bc44c7583" (UID: "9a03da44-5369-4691-9de4-b62bc44c7583"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:15.592261 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592226    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8gh\" (UniqueName: \"kubernetes.io/projected/b19253ac-5ba2-44b0-96d5-664a647032b5-kube-api-access-jg8gh\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592261 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592256    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-util\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592270    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rvd\" (UniqueName: \"kubernetes.io/projected/db23328d-b4c2-4578-a7bf-9b2a11bb8038-kube-api-access-j8rvd\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592282    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592294    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58g8m\" (UniqueName: \"kubernetes.io/projected/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-kube-api-access-58g8m\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592304    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592314    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db23328d-b4c2-4578-a7bf-9b2a11bb8038-util\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592324    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxqtk\" (UniqueName: \"kubernetes.io/projected/9a03da44-5369-4691-9de4-b62bc44c7583-kube-api-access-dxqtk\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592334    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592345    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b19253ac-5ba2-44b0-96d5-664a647032b5-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592354    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a03da44-5369-4691-9de4-b62bc44c7583-util\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:15.592473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:15.592363    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b78ab0-55ce-4b85-b659-fd93a0b5ec7c-util\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:16.229949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.229861    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj"
Sep 09 03:15:16.229949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.229872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnxzj" event={"ID":"93b78ab0-55ce-4b85-b659-fd93a0b5ec7c","Type":"ContainerDied","Data":"ca2bbb9cf33896c4ac71e66e340569728687ec7697293afbb72fb2f888c973e9"}
Sep 09 03:15:16.229949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.229921    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2bbb9cf33896c4ac71e66e340569728687ec7697293afbb72fb2f888c973e9"
Sep 09 03:15:16.231960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.231937    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f"
Sep 09 03:15:16.232130 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.232011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chs62f" event={"ID":"b19253ac-5ba2-44b0-96d5-664a647032b5","Type":"ContainerDied","Data":"f7437b455ea64c1caeb487bd826c1686edec0f6a8afe6495467f7e5f71dfd59d"}
Sep 09 03:15:16.232130 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.232056    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7437b455ea64c1caeb487bd826c1686edec0f6a8afe6495467f7e5f71dfd59d"
Sep 09 03:15:16.234241 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.234215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9" event={"ID":"9a03da44-5369-4691-9de4-b62bc44c7583","Type":"ContainerDied","Data":"7324e8cb9ade01b0d5172effd58031362fc310e64d9d83825e0c9660fb947658"}
Sep 09 03:15:16.234373 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.234259    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7324e8cb9ade01b0d5172effd58031362fc310e64d9d83825e0c9660fb947658"
Sep 09 03:15:16.234373 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.234319    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az86w9"
Sep 09 03:15:16.236716 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.236696    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh" event={"ID":"db23328d-b4c2-4578-a7bf-9b2a11bb8038","Type":"ContainerDied","Data":"105c83906ac7f71bc473bfde768ef98abe19586047459ccd649c390ea8f67763"}
Sep 09 03:15:16.236825 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.236725    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="105c83906ac7f71bc473bfde768ef98abe19586047459ccd649c390ea8f67763"
Sep 09 03:15:16.236825 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:16.236726    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vkblh"
Sep 09 03:15:19.058778 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:19.058739    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:19.853117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:19.853071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:20.652012 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:20.651977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:21.868929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:21.868873    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:21.869423 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:21.868964    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:21.874278 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:21.874251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:22.271778 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:22.271695    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d9578b6c7-zmgw2"
Sep 09 03:15:22.316283 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:22.316247    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c754958-dw2kp"]
Sep 09 03:15:22.417889 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:22.417857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:22.517872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:22.517838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:22.578398 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:22.578366    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:23.392839 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.392803    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"]
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393126    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerName="pull"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393138    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerName="pull"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393148    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a03da44-5369-4691-9de4-b62bc44c7583" containerName="extract"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393153    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a03da44-5369-4691-9de4-b62bc44c7583" containerName="extract"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393162    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerName="extract"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393169    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerName="extract"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393176    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerName="extract"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393181    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerName="extract"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393188    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerName="util"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393194    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerName="util"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393201    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerName="extract"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393206    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerName="extract"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393213    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerName="pull"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393218    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerName="pull"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393225    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerName="util"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393230    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerName="util"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393239    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerName="pull"
Sep 09 03:15:23.393235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393243    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerName="pull"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393249    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerName="util"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393254    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerName="util"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393261    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a03da44-5369-4691-9de4-b62bc44c7583" containerName="util"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393266    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a03da44-5369-4691-9de4-b62bc44c7583" containerName="util"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393273    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a03da44-5369-4691-9de4-b62bc44c7583" containerName="pull"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393277    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a03da44-5369-4691-9de4-b62bc44c7583" containerName="pull"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393348    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a03da44-5369-4691-9de4-b62bc44c7583" containerName="extract"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393354    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b78ab0-55ce-4b85-b659-fd93a0b5ec7c" containerName="extract"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393362    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="db23328d-b4c2-4578-a7bf-9b2a11bb8038" containerName="extract"
Sep 09 03:15:23.394099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.393368    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b19253ac-5ba2-44b0-96d5-664a647032b5" containerName="extract"
Sep 09 03:15:23.397844 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.397816    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:23.399939 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.399917    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 03:15:23.400483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.400468    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p7lmw\""
Sep 09 03:15:23.400553 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.400472    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 03:15:23.408033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.408007    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"]
Sep 09 03:15:23.559953 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.559905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/caefaf43-54b7-47c2-99ab-45bf1951a93a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" (UID: \"caefaf43-54b7-47c2-99ab-45bf1951a93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:23.559953 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.559952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7f8\" (UniqueName: \"kubernetes.io/projected/caefaf43-54b7-47c2-99ab-45bf1951a93a-kube-api-access-gf7f8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" (UID: \"caefaf43-54b7-47c2-99ab-45bf1951a93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:23.660729 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.660623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/caefaf43-54b7-47c2-99ab-45bf1951a93a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" (UID: \"caefaf43-54b7-47c2-99ab-45bf1951a93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:23.660729 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.660675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7f8\" (UniqueName: \"kubernetes.io/projected/caefaf43-54b7-47c2-99ab-45bf1951a93a-kube-api-access-gf7f8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" (UID: \"caefaf43-54b7-47c2-99ab-45bf1951a93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:23.661010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.660990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/caefaf43-54b7-47c2-99ab-45bf1951a93a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" (UID: \"caefaf43-54b7-47c2-99ab-45bf1951a93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:23.673114 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.673087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7f8\" (UniqueName: \"kubernetes.io/projected/caefaf43-54b7-47c2-99ab-45bf1951a93a-kube-api-access-gf7f8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" (UID: \"caefaf43-54b7-47c2-99ab-45bf1951a93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:23.709985 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.709943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:23.990753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:23.990666    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:24.009292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:24.009265    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"]
Sep 09 03:15:24.011826 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:24.011780    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaefaf43_54b7_47c2_99ab_45bf1951a93a.slice/crio-35ca198d34c2e1618964a42240c1407babbbad1e77c5298a3bb9fc0111137361 WatchSource:0}: Error finding container 35ca198d34c2e1618964a42240c1407babbbad1e77c5298a3bb9fc0111137361: Status 404 returned error can't find the container with id 35ca198d34c2e1618964a42240c1407babbbad1e77c5298a3bb9fc0111137361
Sep 09 03:15:24.084567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:24.084516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:24.204873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:24.204823    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:24.279317 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:24.279238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg" event={"ID":"caefaf43-54b7-47c2-99ab-45bf1951a93a","Type":"ContainerStarted","Data":"35ca198d34c2e1618964a42240c1407babbbad1e77c5298a3bb9fc0111137361"}
Sep 09 03:15:25.408164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:25.408122    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"]
Sep 09 03:15:25.411770 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:25.411749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:25.414349 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:25.414319    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9rs2d\""
Sep 09 03:15:25.420425 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:25.420396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"]
Sep 09 03:15:25.580298 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:25.580215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22488\" (UniqueName: \"kubernetes.io/projected/6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a-kube-api-access-22488\") pod \"limitador-operator-controller-manager-74495c69f8-ld6qm\" (UID: \"6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:25.680980 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:25.680886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22488\" (UniqueName: \"kubernetes.io/projected/6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a-kube-api-access-22488\") pod \"limitador-operator-controller-manager-74495c69f8-ld6qm\" (UID: \"6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:25.696637 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:25.696606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22488\" (UniqueName: \"kubernetes.io/projected/6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a-kube-api-access-22488\") pod \"limitador-operator-controller-manager-74495c69f8-ld6qm\" (UID: \"6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:25.728709 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:25.728664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:26.067616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:26.063284    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:26.325772 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:26.325669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:26.753631 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:26.753605    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"]
Sep 09 03:15:26.755447 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:26.755419    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc3c339_01e4_4eb3_b201_9f5a9a0d1a0a.slice/crio-c459857236c5bb0febeb40af9bd09917f8807b5a9316b6d49693c6a8f4fa1284 WatchSource:0}: Error finding container c459857236c5bb0febeb40af9bd09917f8807b5a9316b6d49693c6a8f4fa1284: Status 404 returned error can't find the container with id c459857236c5bb0febeb40af9bd09917f8807b5a9316b6d49693c6a8f4fa1284
Sep 09 03:15:27.301481 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.301446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm" event={"ID":"6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a","Type":"ContainerStarted","Data":"c459857236c5bb0febeb40af9bd09917f8807b5a9316b6d49693c6a8f4fa1284"}
Sep 09 03:15:27.657779 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.657666    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"]
Sep 09 03:15:27.661842 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.661799    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"
Sep 09 03:15:27.664464 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.664161    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8fvw6\""
Sep 09 03:15:27.664464 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.664162    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 03:15:27.676238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.675668    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"]
Sep 09 03:15:27.742492 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.742446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:27.801778 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.801744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jp2\" (UniqueName: \"kubernetes.io/projected/6d7aad25-ad18-4c7e-9c87-f07150f0f0c0-kube-api-access-b7jp2\") pod \"dns-operator-controller-manager-7b84cd7c9c-8zs4k\" (UID: \"6d7aad25-ad18-4c7e-9c87-f07150f0f0c0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"
Sep 09 03:15:27.862700 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.862522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:27.903218 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.903155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jp2\" (UniqueName: \"kubernetes.io/projected/6d7aad25-ad18-4c7e-9c87-f07150f0f0c0-kube-api-access-b7jp2\") pod \"dns-operator-controller-manager-7b84cd7c9c-8zs4k\" (UID: \"6d7aad25-ad18-4c7e-9c87-f07150f0f0c0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"
Sep 09 03:15:27.911934 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.911858    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jp2\" (UniqueName: \"kubernetes.io/projected/6d7aad25-ad18-4c7e-9c87-f07150f0f0c0-kube-api-access-b7jp2\") pod \"dns-operator-controller-manager-7b84cd7c9c-8zs4k\" (UID: \"6d7aad25-ad18-4c7e-9c87-f07150f0f0c0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"
Sep 09 03:15:27.976617 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:27.976583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"
Sep 09 03:15:28.456530 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:28.456487    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:28.978310 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:28.978226    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"]
Sep 09 03:15:28.981569 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:28.981511    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7aad25_ad18_4c7e_9c87_f07150f0f0c0.slice/crio-ecf634a181caf60cca9c3624606cb8c06a29409f1d4f3231f4216d0dc41214e9 WatchSource:0}: Error finding container ecf634a181caf60cca9c3624606cb8c06a29409f1d4f3231f4216d0dc41214e9: Status 404 returned error can't find the container with id ecf634a181caf60cca9c3624606cb8c06a29409f1d4f3231f4216d0dc41214e9
Sep 09 03:15:29.317266 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:29.317228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k" event={"ID":"6d7aad25-ad18-4c7e-9c87-f07150f0f0c0","Type":"ContainerStarted","Data":"ecf634a181caf60cca9c3624606cb8c06a29409f1d4f3231f4216d0dc41214e9"}
Sep 09 03:15:29.319457 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:29.319423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg" event={"ID":"caefaf43-54b7-47c2-99ab-45bf1951a93a","Type":"ContainerStarted","Data":"58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588"}
Sep 09 03:15:29.319677 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:29.319657    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:29.341168 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:29.341112    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg" podStartSLOduration=1.570364347 podStartE2EDuration="6.341095309s" totalImagesPullingTime="4.770730962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:29.338354478 +0000 UTC m=+577.900681023" watchObservedRunningTime="2026-09-09 03:15:29.341095309 +0000 UTC m=+577.903421854"
Sep 09 03:15:29.888501 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:29.888408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:30.324965 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.324934    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5ngn8"]
Sep 09 03:15:30.328586 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.328529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm" event={"ID":"6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a","Type":"ContainerStarted","Data":"824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924"}
Sep 09 03:15:30.328726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.328614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8"
Sep 09 03:15:30.330672 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.330649    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-h2cxf\""
Sep 09 03:15:30.350131 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.350090    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5ngn8"]
Sep 09 03:15:30.352823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.352755    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm" podStartSLOduration=1.927713098 podStartE2EDuration="5.352732736s" totalImagesPullingTime="3.425019638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:30.34888788 +0000 UTC m=+578.911214448" watchObservedRunningTime="2026-09-09 03:15:30.352732736 +0000 UTC m=+578.915059297"
Sep 09 03:15:30.425643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.425600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dcz\" (UniqueName: \"kubernetes.io/projected/a2804f10-5673-483a-b801-659f8f230dfe-kube-api-access-89dcz\") pod \"authorino-operator-76b478bd78-5ngn8\" (UID: \"a2804f10-5673-483a-b801-659f8f230dfe\") " pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8"
Sep 09 03:15:30.526738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.526641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dcz\" (UniqueName: \"kubernetes.io/projected/a2804f10-5673-483a-b801-659f8f230dfe-kube-api-access-89dcz\") pod \"authorino-operator-76b478bd78-5ngn8\" (UID: \"a2804f10-5673-483a-b801-659f8f230dfe\") " pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8"
Sep 09 03:15:30.537443 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.537412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dcz\" (UniqueName: \"kubernetes.io/projected/a2804f10-5673-483a-b801-659f8f230dfe-kube-api-access-89dcz\") pod \"authorino-operator-76b478bd78-5ngn8\" (UID: \"a2804f10-5673-483a-b801-659f8f230dfe\") " pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8"
Sep 09 03:15:30.641045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.641009    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8"
Sep 09 03:15:30.936255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:30.936228    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5ngn8"]
Sep 09 03:15:30.938409 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:30.938371    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2804f10_5673_483a_b801_659f8f230dfe.slice/crio-71bb4431b64a6c919f426901ba2ffb8002f21e93da184c081cd978abbb4d5872 WatchSource:0}: Error finding container 71bb4431b64a6c919f426901ba2ffb8002f21e93da184c081cd978abbb4d5872: Status 404 returned error can't find the container with id 71bb4431b64a6c919f426901ba2ffb8002f21e93da184c081cd978abbb4d5872
Sep 09 03:15:31.336068 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.336028    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8" event={"ID":"a2804f10-5673-483a-b801-659f8f230dfe","Type":"ContainerStarted","Data":"71bb4431b64a6c919f426901ba2ffb8002f21e93da184c081cd978abbb4d5872"}
Sep 09 03:15:31.336468 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.336197    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:31.904205 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.904115    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"]
Sep 09 03:15:31.908174 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.908140    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:31.910409 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.910384    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4tmns\""
Sep 09 03:15:31.910624 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.910605    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 03:15:31.910722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.910698    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 03:15:31.913781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.913739    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:31.920075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:31.920050    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"]
Sep 09 03:15:32.040857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.040826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.041252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.041089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdba21fe-37b1-4d6e-8958-f1023571ab1d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.041252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.041130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46rng\" (UniqueName: \"kubernetes.io/projected/bdba21fe-37b1-4d6e-8958-f1023571ab1d-kube-api-access-46rng\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.142530 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.142477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdba21fe-37b1-4d6e-8958-f1023571ab1d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.142728 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.142532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rng\" (UniqueName: \"kubernetes.io/projected/bdba21fe-37b1-4d6e-8958-f1023571ab1d-kube-api-access-46rng\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.142728 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.142642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.142855 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:32.142781    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 03:15:32.142855 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:32.142838    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert podName:bdba21fe-37b1-4d6e-8958-f1023571ab1d nodeName:}" failed. No retries permitted until 2026-09-09 03:15:32.642816522 +0000 UTC m=+581.205143046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-llm7l" (UID: "bdba21fe-37b1-4d6e-8958-f1023571ab1d") : secret "plugin-serving-cert" not found
Sep 09 03:15:32.143387 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.143350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdba21fe-37b1-4d6e-8958-f1023571ab1d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.152723 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.152692    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rng\" (UniqueName: \"kubernetes.io/projected/bdba21fe-37b1-4d6e-8958-f1023571ab1d-kube-api-access-46rng\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.343910 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.343805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k" event={"ID":"6d7aad25-ad18-4c7e-9c87-f07150f0f0c0","Type":"ContainerStarted","Data":"8293698c32e97696919cf8f65d9b0b9172adfba193fc74158e15d25942064662"}
Sep 09 03:15:32.344394 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.344002    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"
Sep 09 03:15:32.363318 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.363250    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k" podStartSLOduration=2.599035453 podStartE2EDuration="5.363227003s" totalImagesPullingTime="2.76419155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:32.360274572 +0000 UTC m=+580.922601117" watchObservedRunningTime="2026-09-09 03:15:32.363227003 +0000 UTC m=+580.925553551"
Sep 09 03:15:32.647721 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.647626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:32.647878 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:32.647771    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 03:15:32.647878 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:32.647841    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert podName:bdba21fe-37b1-4d6e-8958-f1023571ab1d nodeName:}" failed. No retries permitted until 2026-09-09 03:15:33.647825489 +0000 UTC m=+582.210152011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-llm7l" (UID: "bdba21fe-37b1-4d6e-8958-f1023571ab1d") : secret "plugin-serving-cert" not found
Sep 09 03:15:32.795782 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:32.795741    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:33.350772 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:33.350729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8" event={"ID":"a2804f10-5673-483a-b801-659f8f230dfe","Type":"ContainerStarted","Data":"dac54151a832a63d64aabb31e3b27990360ad5bb2c3036be9f1c475a4a0f5e98"}
Sep 09 03:15:33.351255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:33.350831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8"
Sep 09 03:15:33.369551 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:33.369496    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8" podStartSLOduration=1.643458809 podStartE2EDuration="3.369476328s" totalImagesPullingTime="1.726017519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:33.368015426 +0000 UTC m=+581.930341985" watchObservedRunningTime="2026-09-09 03:15:33.369476328 +0000 UTC m=+581.931802873"
Sep 09 03:15:33.658841 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:33.658747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:33.661435 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:33.661405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdba21fe-37b1-4d6e-8958-f1023571ab1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-llm7l\" (UID: \"bdba21fe-37b1-4d6e-8958-f1023571ab1d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:33.721957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:33.721911    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"
Sep 09 03:15:33.997898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:33.997873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l"]
Sep 09 03:15:34.000375 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:34.000328    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdba21fe_37b1_4d6e_8958_f1023571ab1d.slice/crio-7f90bd90ff092f749a874446696c9263b4b02898d9cfbb860f375b14395cc30a WatchSource:0}: Error finding container 7f90bd90ff092f749a874446696c9263b4b02898d9cfbb860f375b14395cc30a: Status 404 returned error can't find the container with id 7f90bd90ff092f749a874446696c9263b4b02898d9cfbb860f375b14395cc30a
Sep 09 03:15:34.254762 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:34.254670    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:34.311419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:34.311382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:34.357248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:34.357213    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l" event={"ID":"bdba21fe-37b1-4d6e-8958-f1023571ab1d","Type":"ContainerStarted","Data":"7f90bd90ff092f749a874446696c9263b4b02898d9cfbb860f375b14395cc30a"}
Sep 09 03:15:34.651134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:34.651102    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:34.709936 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:34.709906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:35.057558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:35.057494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:35.133274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:35.133222    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:39.396512 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:39.396479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l" event={"ID":"bdba21fe-37b1-4d6e-8958-f1023571ab1d","Type":"ContainerStarted","Data":"dd2938409660ded6bd1cdf35c2da87e438e37eaa22b36fe76a697fe92b88251c"}
Sep 09 03:15:39.412383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:39.412331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-llm7l" podStartSLOduration=3.684793967 podStartE2EDuration="8.412316187s" totalImagesPullingTime="4.72752222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:39.410051652 +0000 UTC m=+587.972378196" watchObservedRunningTime="2026-09-09 03:15:39.412316187 +0000 UTC m=+587.974642731"
Sep 09 03:15:40.328950 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:40.328920    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:41.318815 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:41.318774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:41.427878 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:41.427837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:42.346765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.346732    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:42.441027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.440994    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"]
Sep 09 03:15:42.444840 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.444814    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:42.474295 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.474264    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"]
Sep 09 03:15:42.534718 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.534686    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee57c298-56bb-431f-a14c-b924bd6c60d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" (UID: \"ee57c298-56bb-431f-a14c-b924bd6c60d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:42.534912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.534735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvt8\" (UniqueName: \"kubernetes.io/projected/ee57c298-56bb-431f-a14c-b924bd6c60d8-kube-api-access-rjvt8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" (UID: \"ee57c298-56bb-431f-a14c-b924bd6c60d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:42.551486 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.551447    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"]
Sep 09 03:15:42.551754 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.551735    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-rjvt8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm" podUID="ee57c298-56bb-431f-a14c-b924bd6c60d8"
Sep 09 03:15:42.558239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.558201    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"]
Sep 09 03:15:42.571942 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.571908    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"]
Sep 09 03:15:42.572177 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.572150    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" containerName="manager" containerID="cri-o://58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588" gracePeriod=2
Sep 09 03:15:42.575523 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.575493    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"]
Sep 09 03:15:42.579677 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.579652    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:42.586411 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.586385    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"]
Sep 09 03:15:42.589233 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.589198    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"caefaf43-54b7-47c2-99ab-45bf1951a93a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\": pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:42.590923 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.590889    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:42.602866 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.602787    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"]
Sep 09 03:15:42.606508 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.606481    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"]
Sep 09 03:15:42.607014 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.606991    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" containerName="manager"
Sep 09 03:15:42.607014 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.607016    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" containerName="manager"
Sep 09 03:15:42.607157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.607128    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" containerName="manager"
Sep 09 03:15:42.610108 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.610086    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"]
Sep 09 03:15:42.610235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.610222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:42.610297 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.610256    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" containerName="manager" containerID="cri-o://824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924" gracePeriod=2
Sep 09 03:15:42.618433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.618403    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"]
Sep 09 03:15:42.619088 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.618883    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:42.621090 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.621055    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:42.625272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.625246    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"]
Sep 09 03:15:42.631452 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.631425    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"]
Sep 09 03:15:42.632009 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.631991    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" containerName="manager"
Sep 09 03:15:42.632075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.632012    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" containerName="manager"
Sep 09 03:15:42.632178 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.632166    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" containerName="manager"
Sep 09 03:15:42.635875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.635855    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"
Sep 09 03:15:42.636041 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.636016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2beecdcb-7a46-4930-8f04-634775e98b59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fgvb5\" (UID: \"2beecdcb-7a46-4930-8f04-634775e98b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:42.636100 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.636055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xmn\" (UniqueName: \"kubernetes.io/projected/2beecdcb-7a46-4930-8f04-634775e98b59-kube-api-access-d4xmn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fgvb5\" (UID: \"2beecdcb-7a46-4930-8f04-634775e98b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:42.636148 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.636118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b17c61c-8030-44d4-8ca0-5344cfaafefb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6dl7q\" (UID: \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:42.636198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.636162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee57c298-56bb-431f-a14c-b924bd6c60d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" (UID: \"ee57c298-56bb-431f-a14c-b924bd6c60d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:42.636198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.636190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvt8\" (UniqueName: \"kubernetes.io/projected/ee57c298-56bb-431f-a14c-b924bd6c60d8-kube-api-access-rjvt8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" (UID: \"ee57c298-56bb-431f-a14c-b924bd6c60d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:42.636389 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.636219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9q8\" (UniqueName: \"kubernetes.io/projected/2b17c61c-8030-44d4-8ca0-5344cfaafefb-kube-api-access-vl9q8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6dl7q\" (UID: \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:42.636580 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.636559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee57c298-56bb-431f-a14c-b924bd6c60d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" (UID: \"ee57c298-56bb-431f-a14c-b924bd6c60d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:42.640013 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.639996    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rjvt8 for pod kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm: failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-27vwm" not found
Sep 09 03:15:42.640087 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.640052    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee57c298-56bb-431f-a14c-b924bd6c60d8-kube-api-access-rjvt8 podName:ee57c298-56bb-431f-a14c-b924bd6c60d8 nodeName:}" failed. No retries permitted until 2026-09-09 03:15:43.140036527 +0000 UTC m=+591.702363050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rjvt8" (UniqueName: "kubernetes.io/projected/ee57c298-56bb-431f-a14c-b924bd6c60d8-kube-api-access-rjvt8") pod "kuadrant-operator-controller-manager-5769f99d9f-27vwm" (UID: "ee57c298-56bb-431f-a14c-b924bd6c60d8") : failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-27vwm" not found
Sep 09 03:15:42.647246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.646238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"]
Sep 09 03:15:42.657263 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.657220    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:42.677812 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.677765    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:42.736913 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.736882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrlpm\" (UniqueName: \"kubernetes.io/projected/fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d-kube-api-access-zrlpm\") pod \"limitador-operator-controller-manager-74495c69f8-x4xfd\" (UID: \"fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"
Sep 09 03:15:42.737081 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.736963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b17c61c-8030-44d4-8ca0-5344cfaafefb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6dl7q\" (UID: \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:42.737081 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.737050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9q8\" (UniqueName: \"kubernetes.io/projected/2b17c61c-8030-44d4-8ca0-5344cfaafefb-kube-api-access-vl9q8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6dl7q\" (UID: \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:42.737191 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.737079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2beecdcb-7a46-4930-8f04-634775e98b59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fgvb5\" (UID: \"2beecdcb-7a46-4930-8f04-634775e98b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:42.737191 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.737110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xmn\" (UniqueName: \"kubernetes.io/projected/2beecdcb-7a46-4930-8f04-634775e98b59-kube-api-access-d4xmn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fgvb5\" (UID: \"2beecdcb-7a46-4930-8f04-634775e98b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:42.737396 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.737372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b17c61c-8030-44d4-8ca0-5344cfaafefb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6dl7q\" (UID: \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:42.737515 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.737492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2beecdcb-7a46-4930-8f04-634775e98b59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fgvb5\" (UID: \"2beecdcb-7a46-4930-8f04-634775e98b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:42.749566 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.749501    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xmn\" (UniqueName: \"kubernetes.io/projected/2beecdcb-7a46-4930-8f04-634775e98b59-kube-api-access-d4xmn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fgvb5\" (UID: \"2beecdcb-7a46-4930-8f04-634775e98b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:42.753294 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.753240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9q8\" (UniqueName: \"kubernetes.io/projected/2b17c61c-8030-44d4-8ca0-5344cfaafefb-kube-api-access-vl9q8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6dl7q\" (UID: \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:42.838368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.838335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrlpm\" (UniqueName: \"kubernetes.io/projected/fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d-kube-api-access-zrlpm\") pod \"limitador-operator-controller-manager-74495c69f8-x4xfd\" (UID: \"fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"
Sep 09 03:15:42.847419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.847386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrlpm\" (UniqueName: \"kubernetes.io/projected/fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d-kube-api-access-zrlpm\") pod \"limitador-operator-controller-manager-74495c69f8-x4xfd\" (UID: \"fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"
Sep 09 03:15:42.859787 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.859758    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:42.861634 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.861608    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:42.861818 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.861787    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:42.862995 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.862961    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:42.864404 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.864380    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:42.865700 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:42.865679    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:42.939235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.939196    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22488\" (UniqueName: \"kubernetes.io/projected/6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a-kube-api-access-22488\") pod \"6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a\" (UID: \"6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a\") "
Sep 09 03:15:42.939416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.939274    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/caefaf43-54b7-47c2-99ab-45bf1951a93a-extensions-socket-volume\") pod \"caefaf43-54b7-47c2-99ab-45bf1951a93a\" (UID: \"caefaf43-54b7-47c2-99ab-45bf1951a93a\") "
Sep 09 03:15:42.939416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.939312    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf7f8\" (UniqueName: \"kubernetes.io/projected/caefaf43-54b7-47c2-99ab-45bf1951a93a-kube-api-access-gf7f8\") pod \"caefaf43-54b7-47c2-99ab-45bf1951a93a\" (UID: \"caefaf43-54b7-47c2-99ab-45bf1951a93a\") "
Sep 09 03:15:42.939800 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.939769    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caefaf43-54b7-47c2-99ab-45bf1951a93a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "caefaf43-54b7-47c2-99ab-45bf1951a93a" (UID: "caefaf43-54b7-47c2-99ab-45bf1951a93a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:42.941735 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.941705    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a-kube-api-access-22488" (OuterVolumeSpecName: "kube-api-access-22488") pod "6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" (UID: "6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a"). InnerVolumeSpecName "kube-api-access-22488". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:15:42.941818 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.941751    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caefaf43-54b7-47c2-99ab-45bf1951a93a-kube-api-access-gf7f8" (OuterVolumeSpecName: "kube-api-access-gf7f8") pod "caefaf43-54b7-47c2-99ab-45bf1951a93a" (UID: "caefaf43-54b7-47c2-99ab-45bf1951a93a"). InnerVolumeSpecName "kube-api-access-gf7f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:15:42.989523 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.989489    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:42.996704 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:42.996683    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:43.003460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.003438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"
Sep 09 03:15:43.040958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.040918    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22488\" (UniqueName: \"kubernetes.io/projected/6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a-kube-api-access-22488\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:43.040958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.040957    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/caefaf43-54b7-47c2-99ab-45bf1951a93a-extensions-socket-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:43.041228 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.040973    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7f8\" (UniqueName: \"kubernetes.io/projected/caefaf43-54b7-47c2-99ab-45bf1951a93a-kube-api-access-gf7f8\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:43.141430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.141395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvt8\" (UniqueName: \"kubernetes.io/projected/ee57c298-56bb-431f-a14c-b924bd6c60d8-kube-api-access-rjvt8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" (UID: \"ee57c298-56bb-431f-a14c-b924bd6c60d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:43.143701 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.143681    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rjvt8 for pod kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm: failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-27vwm" not found
Sep 09 03:15:43.143799 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.143747    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee57c298-56bb-431f-a14c-b924bd6c60d8-kube-api-access-rjvt8 podName:ee57c298-56bb-431f-a14c-b924bd6c60d8 nodeName:}" failed. No retries permitted until 2026-09-09 03:15:44.143731366 +0000 UTC m=+592.706057889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rjvt8" (UniqueName: "kubernetes.io/projected/ee57c298-56bb-431f-a14c-b924bd6c60d8-kube-api-access-rjvt8") pod "kuadrant-operator-controller-manager-5769f99d9f-27vwm" (UID: "ee57c298-56bb-431f-a14c-b924bd6c60d8") : failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-27vwm" not found
Sep 09 03:15:43.281041 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.281012    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"]
Sep 09 03:15:43.283616 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:43.283577    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b17c61c_8030_44d4_8ca0_5344cfaafefb.slice/crio-14a1d5bed109d4e0135693b8dc23e25b7721166c77391c051722626ab61bfed3 WatchSource:0}: Error finding container 14a1d5bed109d4e0135693b8dc23e25b7721166c77391c051722626ab61bfed3: Status 404 returned error can't find the container with id 14a1d5bed109d4e0135693b8dc23e25b7721166c77391c051722626ab61bfed3
Sep 09 03:15:43.285798 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.285749    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"]
Sep 09 03:15:43.289303 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:43.289280    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2beecdcb_7a46_4930_8f04_634775e98b59.slice/crio-4ce9566f49a6eeebf1fd9a7a17fb0bec177020163f7f14c34626177ecbb512ed WatchSource:0}: Error finding container 4ce9566f49a6eeebf1fd9a7a17fb0bec177020163f7f14c34626177ecbb512ed: Status 404 returned error can't find the container with id 4ce9566f49a6eeebf1fd9a7a17fb0bec177020163f7f14c34626177ecbb512ed
Sep 09 03:15:43.300079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.300059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"]
Sep 09 03:15:43.302811 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:15:43.302786    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0b58c2_6582_4cd1_90e3_b61b9fbfdd3d.slice/crio-dfc1158967620e33d868bcc1b6b51026b75b25afcbb73fa8ce774c274fdf489e WatchSource:0}: Error finding container dfc1158967620e33d868bcc1b6b51026b75b25afcbb73fa8ce774c274fdf489e: Status 404 returned error can't find the container with id dfc1158967620e33d868bcc1b6b51026b75b25afcbb73fa8ce774c274fdf489e
Sep 09 03:15:43.352963 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.352933    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8zs4k"
Sep 09 03:15:43.354998 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.354806    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:43.357187 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.356412    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:43.426594 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.426532    2562 generic.go:356] "Generic (PLEG): container finished" podID="caefaf43-54b7-47c2-99ab-45bf1951a93a" containerID="58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588" exitCode=0
Sep 09 03:15:43.426594 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.426597    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:43.426863 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.426601    2562 scope.go:117] "RemoveContainer" containerID="58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588"
Sep 09 03:15:43.428460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.428338    2562 generic.go:356] "Generic (PLEG): container finished" podID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" containerID="824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924" exitCode=0
Sep 09 03:15:43.428460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.428449    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:43.428748 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.428725    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:43.430345 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.430309    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:43.431134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.431106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd" event={"ID":"fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d","Type":"ContainerStarted","Data":"ae7076aebf99b23c78fb69566b7563e12dc093a383be7d64e15c85e8178b8fc9"}
Sep 09 03:15:43.431280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.431163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd" event={"ID":"fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d","Type":"ContainerStarted","Data":"dfc1158967620e33d868bcc1b6b51026b75b25afcbb73fa8ce774c274fdf489e"}
Sep 09 03:15:43.431280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.431203    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"
Sep 09 03:15:43.431872 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.431841    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:43.433258 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.433233    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:43.433906 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.433880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q" event={"ID":"2b17c61c-8030-44d4-8ca0-5344cfaafefb","Type":"ContainerStarted","Data":"da5cb2ac05c7dfbabee823cae21eee0974411ff3f7a4f208c933ef6e9d0f762b"}
Sep 09 03:15:43.434006 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.433931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q" event={"ID":"2b17c61c-8030-44d4-8ca0-5344cfaafefb","Type":"ContainerStarted","Data":"14a1d5bed109d4e0135693b8dc23e25b7721166c77391c051722626ab61bfed3"}
Sep 09 03:15:43.434006 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.433959    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:43.434880 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.434857    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:43.436059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.436040    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:43.436252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.436056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5" event={"ID":"2beecdcb-7a46-4930-8f04-634775e98b59","Type":"ContainerStarted","Data":"ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2"}
Sep 09 03:15:43.436329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.436289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5" event={"ID":"2beecdcb-7a46-4930-8f04-634775e98b59","Type":"ContainerStarted","Data":"4ce9566f49a6eeebf1fd9a7a17fb0bec177020163f7f14c34626177ecbb512ed"}
Sep 09 03:15:43.436435 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.436421    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:43.438825 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.438804    2562 scope.go:117] "RemoveContainer" containerID="58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588"
Sep 09 03:15:43.439148 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.439130    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588\": container with ID starting with 58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588 not found: ID does not exist" containerID="58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588"
Sep 09 03:15:43.439209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.439157    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588"} err="failed to get container status \"58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588\": rpc error: code = NotFound desc = could not find container \"58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588\": container with ID starting with 58c9f3dc162088614865c3d3b0ad00f7c3b51efa33f22644e3a5abe7ca84f588 not found: ID does not exist"
Sep 09 03:15:43.439209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.439175    2562 scope.go:117] "RemoveContainer" containerID="824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924"
Sep 09 03:15:43.452887 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.452860    2562 scope.go:117] "RemoveContainer" containerID="824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924"
Sep 09 03:15:43.453098 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.453064    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:43.453702 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.453370    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924\": container with ID starting with 824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924 not found: ID does not exist" containerID="824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924"
Sep 09 03:15:43.453702 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.453453    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924"} err="failed to get container status \"824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924\": rpc error: code = NotFound desc = could not find container \"824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924\": container with ID starting with 824fd9aac49c9b027fd20a0ee70cb094df4373c4016a937830100d6dd4207924 not found: ID does not exist"
Sep 09 03:15:43.453914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.453748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:43.476396 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.476275    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd" podStartSLOduration=1.4762519680000001 podStartE2EDuration="1.476251968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:43.451244166 +0000 UTC m=+592.013570733" watchObservedRunningTime="2026-09-09 03:15:43.476251968 +0000 UTC m=+592.038578534"
Sep 09 03:15:43.477366 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.477331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q" podStartSLOduration=1.4773205520000001 podStartE2EDuration="1.477320552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:43.476080787 +0000 UTC m=+592.038407332" watchObservedRunningTime="2026-09-09 03:15:43.477320552 +0000 UTC m=+592.039647097"
Sep 09 03:15:43.477725 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.477697    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:43.494659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.494613    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5" podStartSLOduration=1.494570845 podStartE2EDuration="1.494570845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:15:43.493978233 +0000 UTC m=+592.056304781" watchObservedRunningTime="2026-09-09 03:15:43.494570845 +0000 UTC m=+592.056897402"
Sep 09 03:15:43.495636 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.495607    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:43.497106 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.497084    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="ee57c298-56bb-431f-a14c-b924bd6c60d8" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:43.498422 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.498397    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dkfsg\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dkfsg"
Sep 09 03:15:43.499787 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.499749    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="ee57c298-56bb-431f-a14c-b924bd6c60d8" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:43.501188 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:43.501164    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ld6qm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ld6qm"
Sep 09 03:15:43.546006 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.545959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee57c298-56bb-431f-a14c-b924bd6c60d8-extensions-socket-volume\") pod \"ee57c298-56bb-431f-a14c-b924bd6c60d8\" (UID: \"ee57c298-56bb-431f-a14c-b924bd6c60d8\") "
Sep 09 03:15:43.546267 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.546249    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjvt8\" (UniqueName: \"kubernetes.io/projected/ee57c298-56bb-431f-a14c-b924bd6c60d8-kube-api-access-rjvt8\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:43.546341 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.546266    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee57c298-56bb-431f-a14c-b924bd6c60d8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee57c298-56bb-431f-a14c-b924bd6c60d8" (UID: "ee57c298-56bb-431f-a14c-b924bd6c60d8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:43.647009 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.646900    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee57c298-56bb-431f-a14c-b924bd6c60d8-extensions-socket-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:43.746378 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.746211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:43.847137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.847100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:43.939005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.938924    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a" path="/var/lib/kubelet/pods/6dc3c339-01e4-4eb3-b201-9f5a9a0d1a0a/volumes"
Sep 09 03:15:43.939266 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.939253    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caefaf43-54b7-47c2-99ab-45bf1951a93a" path="/var/lib/kubelet/pods/caefaf43-54b7-47c2-99ab-45bf1951a93a/volumes"
Sep 09 03:15:43.939531 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:43.939520    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee57c298-56bb-431f-a14c-b924bd6c60d8" path="/var/lib/kubelet/pods/ee57c298-56bb-431f-a14c-b924bd6c60d8/volumes"
Sep 09 03:15:44.359789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:44.359756    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-5ngn8"
Sep 09 03:15:44.446428 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:44.446266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:44.455945 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:44.452811    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-27vwm\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="ee57c298-56bb-431f-a14c-b924bd6c60d8" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-27vwm"
Sep 09 03:15:47.341395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.341349    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78c754958-dw2kp" podUID="7feebf7b-e074-4367-9994-018df06bf800" containerName="console" containerID="cri-o://0364719346c88c68fe5b9f152e3c7128fc7901e32fca1e4fc13aaadd76a93608" gracePeriod=15
Sep 09 03:15:47.466654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.466622    2562 generic.go:356] "Generic (PLEG): container finished" podID="7feebf7b-e074-4367-9994-018df06bf800" containerID="0364719346c88c68fe5b9f152e3c7128fc7901e32fca1e4fc13aaadd76a93608" exitCode=0
Sep 09 03:15:47.466774 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.466692    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c754958-dw2kp" event={"ID":"7feebf7b-e074-4367-9994-018df06bf800","Type":"ContainerDied","Data":"0364719346c88c68fe5b9f152e3c7128fc7901e32fca1e4fc13aaadd76a93608"}
Sep 09 03:15:47.594043 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.593982    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:15:47.687216 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687178    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-oauth-config\") pod \"7feebf7b-e074-4367-9994-018df06bf800\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") "
Sep 09 03:15:47.687401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-serving-cert\") pod \"7feebf7b-e074-4367-9994-018df06bf800\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") "
Sep 09 03:15:47.687401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687262    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-service-ca\") pod \"7feebf7b-e074-4367-9994-018df06bf800\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") "
Sep 09 03:15:47.687401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687281    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-oauth-serving-cert\") pod \"7feebf7b-e074-4367-9994-018df06bf800\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") "
Sep 09 03:15:47.687401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687326    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-trusted-ca-bundle\") pod \"7feebf7b-e074-4367-9994-018df06bf800\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") "
Sep 09 03:15:47.687401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xft9m\" (UniqueName: \"kubernetes.io/projected/7feebf7b-e074-4367-9994-018df06bf800-kube-api-access-xft9m\") pod \"7feebf7b-e074-4367-9994-018df06bf800\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") "
Sep 09 03:15:47.687738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-console-config\") pod \"7feebf7b-e074-4367-9994-018df06bf800\" (UID: \"7feebf7b-e074-4367-9994-018df06bf800\") "
Sep 09 03:15:47.687808 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7feebf7b-e074-4367-9994-018df06bf800" (UID: "7feebf7b-e074-4367-9994-018df06bf800"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:15:47.687808 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687750    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-service-ca" (OuterVolumeSpecName: "service-ca") pod "7feebf7b-e074-4367-9994-018df06bf800" (UID: "7feebf7b-e074-4367-9994-018df06bf800"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:15:47.687933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687902    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7feebf7b-e074-4367-9994-018df06bf800" (UID: "7feebf7b-e074-4367-9994-018df06bf800"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:15:47.688046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.687933    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-console-config" (OuterVolumeSpecName: "console-config") pod "7feebf7b-e074-4367-9994-018df06bf800" (UID: "7feebf7b-e074-4367-9994-018df06bf800"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:15:47.689737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.689713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7feebf7b-e074-4367-9994-018df06bf800-kube-api-access-xft9m" (OuterVolumeSpecName: "kube-api-access-xft9m") pod "7feebf7b-e074-4367-9994-018df06bf800" (UID: "7feebf7b-e074-4367-9994-018df06bf800"). InnerVolumeSpecName "kube-api-access-xft9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:15:47.689857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.689733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7feebf7b-e074-4367-9994-018df06bf800" (UID: "7feebf7b-e074-4367-9994-018df06bf800"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:15:47.689857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.689751    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7feebf7b-e074-4367-9994-018df06bf800" (UID: "7feebf7b-e074-4367-9994-018df06bf800"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:15:47.788494 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.788454    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-serving-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:47.788494 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.788488    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:47.788494 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.788498    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-oauth-serving-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:47.788494 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.788505    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-trusted-ca-bundle\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:47.788829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.788514    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xft9m\" (UniqueName: \"kubernetes.io/projected/7feebf7b-e074-4367-9994-018df06bf800-kube-api-access-xft9m\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:47.788829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.788524    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7feebf7b-e074-4367-9994-018df06bf800-console-config\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:47.788829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:47.788531    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7feebf7b-e074-4367-9994-018df06bf800-console-oauth-config\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:48.473638 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:48.473587    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c754958-dw2kp" event={"ID":"7feebf7b-e074-4367-9994-018df06bf800","Type":"ContainerDied","Data":"d326b741aba15dc2a5a132cc05143e0743291985d7c31ed54acc6f627bc5ac8b"}
Sep 09 03:15:48.474096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:48.473657    2562 scope.go:117] "RemoveContainer" containerID="0364719346c88c68fe5b9f152e3c7128fc7901e32fca1e4fc13aaadd76a93608"
Sep 09 03:15:48.474096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:48.473596    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c754958-dw2kp"
Sep 09 03:15:48.503165 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:48.503129    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c754958-dw2kp"]
Sep 09 03:15:48.510150 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:48.510113    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78c754958-dw2kp"]
Sep 09 03:15:49.938436 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:49.938401    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7feebf7b-e074-4367-9994-018df06bf800" path="/var/lib/kubelet/pods/7feebf7b-e074-4367-9994-018df06bf800/volumes"
Sep 09 03:15:50.287882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:50.287850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:50.388106 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:50.388070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:54.449005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.448973    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-x4xfd"
Sep 09 03:15:54.454533 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.449210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:15:54.524117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.524074    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"]
Sep 09 03:15:54.524384 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.524329    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5" podUID="2beecdcb-7a46-4930-8f04-634775e98b59" containerName="manager" containerID="cri-o://ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2" gracePeriod=10
Sep 09 03:15:54.526641 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.526611    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:54.782196 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.782172    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:54.851250 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.851216    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2beecdcb-7a46-4930-8f04-634775e98b59-extensions-socket-volume\") pod \"2beecdcb-7a46-4930-8f04-634775e98b59\" (UID: \"2beecdcb-7a46-4930-8f04-634775e98b59\") "
Sep 09 03:15:54.851447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.851316    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4xmn\" (UniqueName: \"kubernetes.io/projected/2beecdcb-7a46-4930-8f04-634775e98b59-kube-api-access-d4xmn\") pod \"2beecdcb-7a46-4930-8f04-634775e98b59\" (UID: \"2beecdcb-7a46-4930-8f04-634775e98b59\") "
Sep 09 03:15:54.851706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.851678    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2beecdcb-7a46-4930-8f04-634775e98b59-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2beecdcb-7a46-4930-8f04-634775e98b59" (UID: "2beecdcb-7a46-4930-8f04-634775e98b59"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:15:54.853488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.853471    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2beecdcb-7a46-4930-8f04-634775e98b59-kube-api-access-d4xmn" (OuterVolumeSpecName: "kube-api-access-d4xmn") pod "2beecdcb-7a46-4930-8f04-634775e98b59" (UID: "2beecdcb-7a46-4930-8f04-634775e98b59"). InnerVolumeSpecName "kube-api-access-d4xmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:15:54.952310 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.952275    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2beecdcb-7a46-4930-8f04-634775e98b59-extensions-socket-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:54.952310 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:54.952303    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xmn\" (UniqueName: \"kubernetes.io/projected/2beecdcb-7a46-4930-8f04-634775e98b59-kube-api-access-d4xmn\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:15:55.492625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.492587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:55.520066 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.520028    2562 generic.go:356] "Generic (PLEG): container finished" podID="2beecdcb-7a46-4930-8f04-634775e98b59" containerID="ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2" exitCode=0
Sep 09 03:15:55.520249 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.520079    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"
Sep 09 03:15:55.520249 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.520088    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5" event={"ID":"2beecdcb-7a46-4930-8f04-634775e98b59","Type":"ContainerDied","Data":"ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2"}
Sep 09 03:15:55.520249 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.520145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5" event={"ID":"2beecdcb-7a46-4930-8f04-634775e98b59","Type":"ContainerDied","Data":"4ce9566f49a6eeebf1fd9a7a17fb0bec177020163f7f14c34626177ecbb512ed"}
Sep 09 03:15:55.520249 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.520175    2562 scope.go:117] "RemoveContainer" containerID="ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2"
Sep 09 03:15:55.530024 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.529812    2562 scope.go:117] "RemoveContainer" containerID="ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2"
Sep 09 03:15:55.530138 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:15:55.530115    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2\": container with ID starting with ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2 not found: ID does not exist" containerID="ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2"
Sep 09 03:15:55.530198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.530149    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2"} err="failed to get container status \"ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2\": rpc error: code = NotFound desc = could not find container \"ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2\": container with ID starting with ae8cec63d251579d8cd549c2f8dfb8d633917395d4eec7578033b8a9d95970d2 not found: ID does not exist"
Sep 09 03:15:55.550025 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.549917    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"]
Sep 09 03:15:55.553783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.553758    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fgvb5"]
Sep 09 03:15:55.614751 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.614715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:55.892861 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.892823    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:55.938315 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.938283    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2beecdcb-7a46-4930-8f04-634775e98b59" path="/var/lib/kubelet/pods/2beecdcb-7a46-4930-8f04-634775e98b59/volumes"
Sep 09 03:15:55.992959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:55.992927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:56.094413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:56.094381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:15:56.194022 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:15:56.193927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:01.623877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:01.623837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:01.723618 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:01.723585    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:03.079096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:03.079051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:03.179695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:03.179657    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:05.514366 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:05.514328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:05.615389 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:05.615352    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:10.847570 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.847507    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"]
Sep 09 03:16:10.847937 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.847900    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2beecdcb-7a46-4930-8f04-634775e98b59" containerName="manager"
Sep 09 03:16:10.847937 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.847913    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2beecdcb-7a46-4930-8f04-634775e98b59" containerName="manager"
Sep 09 03:16:10.847937 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.847933    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7feebf7b-e074-4367-9994-018df06bf800" containerName="console"
Sep 09 03:16:10.847937 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.847938    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7feebf7b-e074-4367-9994-018df06bf800" containerName="console"
Sep 09 03:16:10.848085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.847986    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2beecdcb-7a46-4930-8f04-634775e98b59" containerName="manager"
Sep 09 03:16:10.848085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.847999    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7feebf7b-e074-4367-9994-018df06bf800" containerName="console"
Sep 09 03:16:10.865375 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.865340    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"]
Sep 09 03:16:10.865557 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.865448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.867477 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.867455    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 03:16:10.867600 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.867527    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f84hs\""
Sep 09 03:16:10.904710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.904658    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:10.995159 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.995159 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.995414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.995414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvglw\" (UniqueName: \"kubernetes.io/projected/9142fd43-df2e-4dc5-b146-029f01d15c7a-kube-api-access-nvglw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.995414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9142fd43-df2e-4dc5-b146-029f01d15c7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.995414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.995414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.995414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:10.995693 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:10.995449    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.096662 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.096662 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.096933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.096933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.096933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvglw\" (UniqueName: \"kubernetes.io/projected/9142fd43-df2e-4dc5-b146-029f01d15c7a-kube-api-access-nvglw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.096933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9142fd43-df2e-4dc5-b146-029f01d15c7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.096933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.097195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.097195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.096997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.097195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.097099    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.097195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.097159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.097387 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.097297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.097387 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.097355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.097759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.097709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9142fd43-df2e-4dc5-b146-029f01d15c7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.099383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.099360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.099874 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.099858    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.103908 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.103881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9142fd43-df2e-4dc5-b146-029f01d15c7a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.104085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.104066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvglw\" (UniqueName: \"kubernetes.io/projected/9142fd43-df2e-4dc5-b146-029f01d15c7a-kube-api-access-nvglw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zvtkx\" (UID: \"9142fd43-df2e-4dc5-b146-029f01d15c7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.178707 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.178673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:11.424680 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.424587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:11.456494 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.456464    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"]
Sep 09 03:16:11.458405 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:16:11.458377    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9142fd43_df2e_4dc5_b146_029f01d15c7a.slice/crio-3909ff3af6c5a8b34e727d792dc0cefa9492b6e33d10a81a26d36af9216730dd WatchSource:0}: Error finding container 3909ff3af6c5a8b34e727d792dc0cefa9492b6e33d10a81a26d36af9216730dd: Status 404 returned error can't find the container with id 3909ff3af6c5a8b34e727d792dc0cefa9492b6e33d10a81a26d36af9216730dd
Sep 09 03:16:11.605777 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:11.605744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" event={"ID":"9142fd43-df2e-4dc5-b146-029f01d15c7a","Type":"ContainerStarted","Data":"3909ff3af6c5a8b34e727d792dc0cefa9492b6e33d10a81a26d36af9216730dd"}
Sep 09 03:16:12.077788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:12.077753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:12.175758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:12.175721    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:15.029073 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:15.029031    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:16:15.029416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:15.029109    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:16:15.029416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:15.029138    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:16:15.253960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:15.253920    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:15.354625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:15.354588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:15.636970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:15.636882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" event={"ID":"9142fd43-df2e-4dc5-b146-029f01d15c7a","Type":"ContainerStarted","Data":"a0cad858001df230fbc4bddf99e31c638ef33e9996c1fc381a097a99168ec477"}
Sep 09 03:16:15.655933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:15.655883    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" podStartSLOduration=2.087684101 podStartE2EDuration="5.655862333s" totalImagesPullingTime="3.568178232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:15.655183688 +0000 UTC m=+624.217510233" watchObservedRunningTime="2026-09-09 03:16:15.655862333 +0000 UTC m=+624.218188882"
Sep 09 03:16:16.178882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.178836    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:16.183737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.183711    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:16.642029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.641997    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:16.642893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.642873    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:16:16.757348 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.757318    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-j9pfl"]
Sep 09 03:16:16.760237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.760215    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:16.760766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.760741    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:16.762254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.762233    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 03:16:16.768693 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.768668    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-j9pfl"]
Sep 09 03:16:16.855323 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.855280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-j9pfl"]
Sep 09 03:16:16.956060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.955964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10c64e33-5db3-4b0f-9966-fa6f655c03a8-config-file\") pod \"limitador-limitador-8588749977-j9pfl\" (UID: \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\") " pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:16.956250 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:16.956135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj24l\" (UniqueName: \"kubernetes.io/projected/10c64e33-5db3-4b0f-9966-fa6f655c03a8-kube-api-access-zj24l\") pod \"limitador-limitador-8588749977-j9pfl\" (UID: \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\") " pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:17.012955 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.012924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:17.056912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.056876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj24l\" (UniqueName: \"kubernetes.io/projected/10c64e33-5db3-4b0f-9966-fa6f655c03a8-kube-api-access-zj24l\") pod \"limitador-limitador-8588749977-j9pfl\" (UID: \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\") " pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:17.057126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.056931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10c64e33-5db3-4b0f-9966-fa6f655c03a8-config-file\") pod \"limitador-limitador-8588749977-j9pfl\" (UID: \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\") " pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:17.057659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.057639    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10c64e33-5db3-4b0f-9966-fa6f655c03a8-config-file\") pod \"limitador-limitador-8588749977-j9pfl\" (UID: \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\") " pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:17.064584 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.064566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj24l\" (UniqueName: \"kubernetes.io/projected/10c64e33-5db3-4b0f-9966-fa6f655c03a8-kube-api-access-zj24l\") pod \"limitador-limitador-8588749977-j9pfl\" (UID: \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\") " pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:17.073695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.073659    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:17.074795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.074772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:17.343651 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.343623    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-j9pfl"]
Sep 09 03:16:17.344980 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:16:17.344932    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c64e33_5db3_4b0f_9966_fa6f655c03a8.slice/crio-eeadd783b0a0b7a48f53f30ca7d7e3f19d9c929a076a5e46f7289fc97fc401e9 WatchSource:0}: Error finding container eeadd783b0a0b7a48f53f30ca7d7e3f19d9c929a076a5e46f7289fc97fc401e9: Status 404 returned error can't find the container with id eeadd783b0a0b7a48f53f30ca7d7e3f19d9c929a076a5e46f7289fc97fc401e9
Sep 09 03:16:17.616075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.615972    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:17.649201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.649161    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl" event={"ID":"10c64e33-5db3-4b0f-9966-fa6f655c03a8","Type":"ContainerStarted","Data":"eeadd783b0a0b7a48f53f30ca7d7e3f19d9c929a076a5e46f7289fc97fc401e9"}
Sep 09 03:16:17.683328 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.683284    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:17.683958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.683939    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-fplwj"]
Sep 09 03:16:17.687068 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.687047    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fplwj"
Sep 09 03:16:17.689080 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.689059    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l5zrp\""
Sep 09 03:16:17.698144 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.698105    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fplwj"]
Sep 09 03:16:17.864374 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.864331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtp9f\" (UniqueName: \"kubernetes.io/projected/290b4a4f-433f-43a0-9026-cc9d100e47af-kube-api-access-jtp9f\") pod \"authorino-59c974bd48-fplwj\" (UID: \"290b4a4f-433f-43a0-9026-cc9d100e47af\") " pod="kuadrant-system/authorino-59c974bd48-fplwj"
Sep 09 03:16:17.965570 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.965471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtp9f\" (UniqueName: \"kubernetes.io/projected/290b4a4f-433f-43a0-9026-cc9d100e47af-kube-api-access-jtp9f\") pod \"authorino-59c974bd48-fplwj\" (UID: \"290b4a4f-433f-43a0-9026-cc9d100e47af\") " pod="kuadrant-system/authorino-59c974bd48-fplwj"
Sep 09 03:16:17.973274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.973235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtp9f\" (UniqueName: \"kubernetes.io/projected/290b4a4f-433f-43a0-9026-cc9d100e47af-kube-api-access-jtp9f\") pod \"authorino-59c974bd48-fplwj\" (UID: \"290b4a4f-433f-43a0-9026-cc9d100e47af\") " pod="kuadrant-system/authorino-59c974bd48-fplwj"
Sep 09 03:16:17.997693 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:17.997649    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fplwj"
Sep 09 03:16:18.268757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:18.268730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fplwj"]
Sep 09 03:16:18.270885 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:16:18.270851    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod290b4a4f_433f_43a0_9026_cc9d100e47af.slice/crio-dec4e23ce107cd518df4c8b1ae5efb9edeeccfff3234ff26f4900b595f983111 WatchSource:0}: Error finding container dec4e23ce107cd518df4c8b1ae5efb9edeeccfff3234ff26f4900b595f983111: Status 404 returned error can't find the container with id dec4e23ce107cd518df4c8b1ae5efb9edeeccfff3234ff26f4900b595f983111
Sep 09 03:16:18.659886 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:18.659853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fplwj" event={"ID":"290b4a4f-433f-43a0-9026-cc9d100e47af","Type":"ContainerStarted","Data":"dec4e23ce107cd518df4c8b1ae5efb9edeeccfff3234ff26f4900b595f983111"}
Sep 09 03:16:19.668581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:19.668518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl" event={"ID":"10c64e33-5db3-4b0f-9966-fa6f655c03a8","Type":"ContainerStarted","Data":"06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285"}
Sep 09 03:16:19.669066 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:19.668690    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:19.683496 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:19.683442    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl" podStartSLOduration=2.337064185 podStartE2EDuration="3.683425842s" totalImagesPullingTime="1.346361657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:19.682442005 +0000 UTC m=+628.244768550" watchObservedRunningTime="2026-09-09 03:16:19.683425842 +0000 UTC m=+628.245752399"
Sep 09 03:16:20.675803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:20.675762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fplwj" event={"ID":"290b4a4f-433f-43a0-9026-cc9d100e47af","Type":"ContainerStarted","Data":"02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e"}
Sep 09 03:16:20.690619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:20.690561    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-fplwj" podStartSLOduration=1.971189648 podStartE2EDuration="3.690529674s" totalImagesPullingTime="1.719340026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:20.688441696 +0000 UTC m=+629.250768245" watchObservedRunningTime="2026-09-09 03:16:20.690529674 +0000 UTC m=+629.252856276"
Sep 09 03:16:30.677212 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:30.677172    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:31.453840 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:31.453797    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-j9pfl"]
Sep 09 03:16:31.454054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:31.454006    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl" podUID="10c64e33-5db3-4b0f-9966-fa6f655c03a8" containerName="limitador" containerID="cri-o://06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285" gracePeriod=30
Sep 09 03:16:32.018866 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.018795    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:32.082280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.082247    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10c64e33-5db3-4b0f-9966-fa6f655c03a8-config-file\") pod \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\" (UID: \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\") "
Sep 09 03:16:32.082488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.082365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj24l\" (UniqueName: \"kubernetes.io/projected/10c64e33-5db3-4b0f-9966-fa6f655c03a8-kube-api-access-zj24l\") pod \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\" (UID: \"10c64e33-5db3-4b0f-9966-fa6f655c03a8\") "
Sep 09 03:16:32.082651 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.082627    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c64e33-5db3-4b0f-9966-fa6f655c03a8-config-file" (OuterVolumeSpecName: "config-file") pod "10c64e33-5db3-4b0f-9966-fa6f655c03a8" (UID: "10c64e33-5db3-4b0f-9966-fa6f655c03a8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:16:32.084700 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.084673    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c64e33-5db3-4b0f-9966-fa6f655c03a8-kube-api-access-zj24l" (OuterVolumeSpecName: "kube-api-access-zj24l") pod "10c64e33-5db3-4b0f-9966-fa6f655c03a8" (UID: "10c64e33-5db3-4b0f-9966-fa6f655c03a8"). InnerVolumeSpecName "kube-api-access-zj24l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:16:32.183444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.183401    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10c64e33-5db3-4b0f-9966-fa6f655c03a8-config-file\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:16:32.183444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.183437    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj24l\" (UniqueName: \"kubernetes.io/projected/10c64e33-5db3-4b0f-9966-fa6f655c03a8-kube-api-access-zj24l\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:16:32.746309 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.746277    2562 generic.go:356] "Generic (PLEG): container finished" podID="10c64e33-5db3-4b0f-9966-fa6f655c03a8" containerID="06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285" exitCode=0
Sep 09 03:16:32.746499 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.746341    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl"
Sep 09 03:16:32.746499 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.746343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl" event={"ID":"10c64e33-5db3-4b0f-9966-fa6f655c03a8","Type":"ContainerDied","Data":"06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285"}
Sep 09 03:16:32.746499 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.746463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-j9pfl" event={"ID":"10c64e33-5db3-4b0f-9966-fa6f655c03a8","Type":"ContainerDied","Data":"eeadd783b0a0b7a48f53f30ca7d7e3f19d9c929a076a5e46f7289fc97fc401e9"}
Sep 09 03:16:32.746499 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.746485    2562 scope.go:117] "RemoveContainer" containerID="06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285"
Sep 09 03:16:32.756399 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.756378    2562 scope.go:117] "RemoveContainer" containerID="06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285"
Sep 09 03:16:32.756739 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:16:32.756720    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285\": container with ID starting with 06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285 not found: ID does not exist" containerID="06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285"
Sep 09 03:16:32.756797 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.756750    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285"} err="failed to get container status \"06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285\": rpc error: code = NotFound desc = could not find container \"06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285\": container with ID starting with 06a1356e6cd50cafc9bb7ad600c8b5d44266ddc0421684d719236a9ea70cd285 not found: ID does not exist"
Sep 09 03:16:32.767401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.767366    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-j9pfl"]
Sep 09 03:16:32.769580 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.769559    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-j9pfl"]
Sep 09 03:16:32.953608 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:32.953572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:33.051319 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.051282    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:33.464960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.464879    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"]
Sep 09 03:16:33.465244 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.465231    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c64e33-5db3-4b0f-9966-fa6f655c03a8" containerName="limitador"
Sep 09 03:16:33.465290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.465245    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c64e33-5db3-4b0f-9966-fa6f655c03a8" containerName="limitador"
Sep 09 03:16:33.465326 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.465317    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c64e33-5db3-4b0f-9966-fa6f655c03a8" containerName="limitador"
Sep 09 03:16:33.469787 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.469770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:33.472142 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.472119    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 03:16:33.472324 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.472291    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:16:33.472862 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.472667    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 03:16:33.472862 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.472702    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-66kgz\""
Sep 09 03:16:33.475285 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.475255    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"]
Sep 09 03:16:33.493889 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.493855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm7mf\" (UniqueName: \"kubernetes.io/projected/81eff2b3-3e9a-4838-ab55-86a00eb3a3f4-kube-api-access-vm7mf\") pod \"postgres-69fb4594cf-ql9d6\" (UID: \"81eff2b3-3e9a-4838-ab55-86a00eb3a3f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:33.494062 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.493940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/81eff2b3-3e9a-4838-ab55-86a00eb3a3f4-data\") pod \"postgres-69fb4594cf-ql9d6\" (UID: \"81eff2b3-3e9a-4838-ab55-86a00eb3a3f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:33.582567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.582497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:33.594660 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.594630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/81eff2b3-3e9a-4838-ab55-86a00eb3a3f4-data\") pod \"postgres-69fb4594cf-ql9d6\" (UID: \"81eff2b3-3e9a-4838-ab55-86a00eb3a3f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:33.594810 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.594694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm7mf\" (UniqueName: \"kubernetes.io/projected/81eff2b3-3e9a-4838-ab55-86a00eb3a3f4-kube-api-access-vm7mf\") pod \"postgres-69fb4594cf-ql9d6\" (UID: \"81eff2b3-3e9a-4838-ab55-86a00eb3a3f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:33.595033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.595015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/81eff2b3-3e9a-4838-ab55-86a00eb3a3f4-data\") pod \"postgres-69fb4594cf-ql9d6\" (UID: \"81eff2b3-3e9a-4838-ab55-86a00eb3a3f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:33.602809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.602781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm7mf\" (UniqueName: \"kubernetes.io/projected/81eff2b3-3e9a-4838-ab55-86a00eb3a3f4-kube-api-access-vm7mf\") pod \"postgres-69fb4594cf-ql9d6\" (UID: \"81eff2b3-3e9a-4838-ab55-86a00eb3a3f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:33.781523 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.781428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:33.939333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:33.939303    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c64e33-5db3-4b0f-9966-fa6f655c03a8" path="/var/lib/kubelet/pods/10c64e33-5db3-4b0f-9966-fa6f655c03a8/volumes"
Sep 09 03:16:34.054929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:34.054889    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"]
Sep 09 03:16:34.058240 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:16:34.058210    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81eff2b3_3e9a_4838_ab55_86a00eb3a3f4.slice/crio-d543580a2a08b28913cdad80aa6f364d88198f6e0b6904093c99b00f3698078f WatchSource:0}: Error finding container d543580a2a08b28913cdad80aa6f364d88198f6e0b6904093c99b00f3698078f: Status 404 returned error can't find the container with id d543580a2a08b28913cdad80aa6f364d88198f6e0b6904093c99b00f3698078f
Sep 09 03:16:34.762013 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:34.761974    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6" event={"ID":"81eff2b3-3e9a-4838-ab55-86a00eb3a3f4","Type":"ContainerStarted","Data":"d543580a2a08b28913cdad80aa6f364d88198f6e0b6904093c99b00f3698078f"}
Sep 09 03:16:36.774766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:36.774670    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6" event={"ID":"81eff2b3-3e9a-4838-ab55-86a00eb3a3f4","Type":"ContainerStarted","Data":"1b515b0c4c647185394232567d4d1c83fb9e7fd54f64db9fdf04bc3edbb9cdcd"}
Sep 09 03:16:36.775189 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:36.774772    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:36.791552 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:36.791478    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6" podStartSLOduration=1.537349564 podStartE2EDuration="3.791459154s" totalImagesPullingTime="2.25410959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:36.787995049 +0000 UTC m=+645.350321596" watchObservedRunningTime="2026-09-09 03:16:36.791459154 +0000 UTC m=+645.353785700"
Sep 09 03:16:42.811480 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:42.811453    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ql9d6"
Sep 09 03:16:43.172968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.172858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:43.302636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.302591    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:43.365672 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.365641    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-5hthg"]
Sep 09 03:16:43.372142 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.372102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5hthg"
Sep 09 03:16:43.375329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.375300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-5hthg"]
Sep 09 03:16:43.489389 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.489308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqg8\" (UniqueName: \"kubernetes.io/projected/578c7cfb-b998-4318-b6db-ce8fa3f30ef5-kube-api-access-xxqg8\") pod \"authorino-557f876f69-5hthg\" (UID: \"578c7cfb-b998-4318-b6db-ce8fa3f30ef5\") " pod="kuadrant-system/authorino-557f876f69-5hthg"
Sep 09 03:16:43.496387 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.496359    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5hthg"]
Sep 09 03:16:43.496629 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:16:43.496612    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xxqg8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-5hthg" podUID="578c7cfb-b998-4318-b6db-ce8fa3f30ef5"
Sep 09 03:16:43.517911 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.517876    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-65ttk"]
Sep 09 03:16:43.521980 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.521963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.524060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.524029    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 03:16:43.530995 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.530931    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-65ttk"]
Sep 09 03:16:43.590288 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.590254    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pww2x\" (UniqueName: \"kubernetes.io/projected/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-kube-api-access-pww2x\") pod \"authorino-6d4dfc649f-65ttk\" (UID: \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\") " pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.590490 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.590320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-tls-cert\") pod \"authorino-6d4dfc649f-65ttk\" (UID: \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\") " pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.590490 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.590440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqg8\" (UniqueName: \"kubernetes.io/projected/578c7cfb-b998-4318-b6db-ce8fa3f30ef5-kube-api-access-xxqg8\") pod \"authorino-557f876f69-5hthg\" (UID: \"578c7cfb-b998-4318-b6db-ce8fa3f30ef5\") " pod="kuadrant-system/authorino-557f876f69-5hthg"
Sep 09 03:16:43.598217 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.598192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqg8\" (UniqueName: \"kubernetes.io/projected/578c7cfb-b998-4318-b6db-ce8fa3f30ef5-kube-api-access-xxqg8\") pod \"authorino-557f876f69-5hthg\" (UID: \"578c7cfb-b998-4318-b6db-ce8fa3f30ef5\") " pod="kuadrant-system/authorino-557f876f69-5hthg"
Sep 09 03:16:43.607594 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.607564    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-65ttk"]
Sep 09 03:16:43.607895 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:16:43.607876    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pww2x tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-65ttk" podUID="4e8ad0b8-61a1-4d05-b785-5a569537a6b4"
Sep 09 03:16:43.633031 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.632992    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cd969b7ff-cdfbs"]
Sep 09 03:16:43.637556 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.637522    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:16:43.642803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.642773    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd969b7ff-cdfbs"]
Sep 09 03:16:43.691385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.691349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-tls-cert\") pod \"authorino-6d4dfc649f-65ttk\" (UID: \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\") " pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.691571 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.691437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pww2x\" (UniqueName: \"kubernetes.io/projected/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-kube-api-access-pww2x\") pod \"authorino-6d4dfc649f-65ttk\" (UID: \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\") " pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.694017 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.693996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-tls-cert\") pod \"authorino-6d4dfc649f-65ttk\" (UID: \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\") " pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.698314 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.698293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pww2x\" (UniqueName: \"kubernetes.io/projected/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-kube-api-access-pww2x\") pod \"authorino-6d4dfc649f-65ttk\" (UID: \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\") " pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.792806 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.792769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98vtk\" (UniqueName: \"kubernetes.io/projected/00cfb66b-52be-4540-8f43-777077c2cb08-kube-api-access-98vtk\") pod \"authorino-6cd969b7ff-cdfbs\" (UID: \"00cfb66b-52be-4540-8f43-777077c2cb08\") " pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:16:43.792967 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.792932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00cfb66b-52be-4540-8f43-777077c2cb08-tls-cert\") pod \"authorino-6cd969b7ff-cdfbs\" (UID: \"00cfb66b-52be-4540-8f43-777077c2cb08\") " pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:16:43.815503 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.815470    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5hthg"
Sep 09 03:16:43.815939 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.815470    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.823676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.823650    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5hthg"
Sep 09 03:16:43.826686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.826667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:43.894341 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.894303    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pww2x\" (UniqueName: \"kubernetes.io/projected/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-kube-api-access-pww2x\") pod \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\" (UID: \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\") "
Sep 09 03:16:43.894529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.894398    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxqg8\" (UniqueName: \"kubernetes.io/projected/578c7cfb-b998-4318-b6db-ce8fa3f30ef5-kube-api-access-xxqg8\") pod \"578c7cfb-b998-4318-b6db-ce8fa3f30ef5\" (UID: \"578c7cfb-b998-4318-b6db-ce8fa3f30ef5\") "
Sep 09 03:16:43.894529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.894430    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-tls-cert\") pod \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\" (UID: \"4e8ad0b8-61a1-4d05-b785-5a569537a6b4\") "
Sep 09 03:16:43.894661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.894534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00cfb66b-52be-4540-8f43-777077c2cb08-tls-cert\") pod \"authorino-6cd969b7ff-cdfbs\" (UID: \"00cfb66b-52be-4540-8f43-777077c2cb08\") " pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:16:43.894716 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.894689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98vtk\" (UniqueName: \"kubernetes.io/projected/00cfb66b-52be-4540-8f43-777077c2cb08-kube-api-access-98vtk\") pod \"authorino-6cd969b7ff-cdfbs\" (UID: \"00cfb66b-52be-4540-8f43-777077c2cb08\") " pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:16:43.896849 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.896819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4e8ad0b8-61a1-4d05-b785-5a569537a6b4" (UID: "4e8ad0b8-61a1-4d05-b785-5a569537a6b4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:16:43.897258 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.897226    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-kube-api-access-pww2x" (OuterVolumeSpecName: "kube-api-access-pww2x") pod "4e8ad0b8-61a1-4d05-b785-5a569537a6b4" (UID: "4e8ad0b8-61a1-4d05-b785-5a569537a6b4"). InnerVolumeSpecName "kube-api-access-pww2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:16:43.897258 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.897243    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00cfb66b-52be-4540-8f43-777077c2cb08-tls-cert\") pod \"authorino-6cd969b7ff-cdfbs\" (UID: \"00cfb66b-52be-4540-8f43-777077c2cb08\") " pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:16:43.897258 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.897262    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578c7cfb-b998-4318-b6db-ce8fa3f30ef5-kube-api-access-xxqg8" (OuterVolumeSpecName: "kube-api-access-xxqg8") pod "578c7cfb-b998-4318-b6db-ce8fa3f30ef5" (UID: "578c7cfb-b998-4318-b6db-ce8fa3f30ef5"). InnerVolumeSpecName "kube-api-access-xxqg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:16:43.902195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.902166    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98vtk\" (UniqueName: \"kubernetes.io/projected/00cfb66b-52be-4540-8f43-777077c2cb08-kube-api-access-98vtk\") pod \"authorino-6cd969b7ff-cdfbs\" (UID: \"00cfb66b-52be-4540-8f43-777077c2cb08\") " pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:16:43.948376 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.948355    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:16:43.995684 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.995654    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqg8\" (UniqueName: \"kubernetes.io/projected/578c7cfb-b998-4318-b6db-ce8fa3f30ef5-kube-api-access-xxqg8\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:16:43.995684 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.995682    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-tls-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:16:43.995684 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:43.995692    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pww2x\" (UniqueName: \"kubernetes.io/projected/4e8ad0b8-61a1-4d05-b785-5a569537a6b4-kube-api-access-pww2x\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:16:44.218430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:44.218403    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd969b7ff-cdfbs"]
Sep 09 03:16:44.220682 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:16:44.220646    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00cfb66b_52be_4540_8f43_777077c2cb08.slice/crio-31d1b579ef3a1e460eb0be7fd4f15d6ea0d6a878a538d13de71f63dc069f9897 WatchSource:0}: Error finding container 31d1b579ef3a1e460eb0be7fd4f15d6ea0d6a878a538d13de71f63dc069f9897: Status 404 returned error can't find the container with id 31d1b579ef3a1e460eb0be7fd4f15d6ea0d6a878a538d13de71f63dc069f9897
Sep 09 03:16:44.822180 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:44.822149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs" event={"ID":"00cfb66b-52be-4540-8f43-777077c2cb08","Type":"ContainerStarted","Data":"31d1b579ef3a1e460eb0be7fd4f15d6ea0d6a878a538d13de71f63dc069f9897"}
Sep 09 03:16:44.822643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:44.822216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5hthg"
Sep 09 03:16:44.822643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:44.822227    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-65ttk"
Sep 09 03:16:44.847158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:44.847128    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-65ttk"]
Sep 09 03:16:44.851090 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:44.851054    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-65ttk"]
Sep 09 03:16:44.878377 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:44.878335    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5hthg"]
Sep 09 03:16:44.881376 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:44.881347    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-5hthg"]
Sep 09 03:16:45.830005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:45.829969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs" event={"ID":"00cfb66b-52be-4540-8f43-777077c2cb08","Type":"ContainerStarted","Data":"6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e"}
Sep 09 03:16:45.847090 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:45.847037    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs" podStartSLOduration=2.167326707 podStartE2EDuration="2.847018525s" totalImagesPullingTime="679.691818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:45.846050724 +0000 UTC m=+654.408377271" watchObservedRunningTime="2026-09-09 03:16:45.847018525 +0000 UTC m=+654.409345070"
Sep 09 03:16:45.873574 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:45.873526    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fplwj"]
Sep 09 03:16:45.873773 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:45.873747    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-fplwj" podUID="290b4a4f-433f-43a0-9026-cc9d100e47af" containerName="authorino" containerID="cri-o://02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e" gracePeriod=30
Sep 09 03:16:45.939525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:45.939494    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e8ad0b8-61a1-4d05-b785-5a569537a6b4" path="/var/lib/kubelet/pods/4e8ad0b8-61a1-4d05-b785-5a569537a6b4/volumes"
Sep 09 03:16:45.939765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:45.939752    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="578c7cfb-b998-4318-b6db-ce8fa3f30ef5" path="/var/lib/kubelet/pods/578c7cfb-b998-4318-b6db-ce8fa3f30ef5/volumes"
Sep 09 03:16:46.129456 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.129426    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fplwj"
Sep 09 03:16:46.214489 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.214450    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtp9f\" (UniqueName: \"kubernetes.io/projected/290b4a4f-433f-43a0-9026-cc9d100e47af-kube-api-access-jtp9f\") pod \"290b4a4f-433f-43a0-9026-cc9d100e47af\" (UID: \"290b4a4f-433f-43a0-9026-cc9d100e47af\") "
Sep 09 03:16:46.216691 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.216666    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290b4a4f-433f-43a0-9026-cc9d100e47af-kube-api-access-jtp9f" (OuterVolumeSpecName: "kube-api-access-jtp9f") pod "290b4a4f-433f-43a0-9026-cc9d100e47af" (UID: "290b4a4f-433f-43a0-9026-cc9d100e47af"). InnerVolumeSpecName "kube-api-access-jtp9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:16:46.315465 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.315429    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtp9f\" (UniqueName: \"kubernetes.io/projected/290b4a4f-433f-43a0-9026-cc9d100e47af-kube-api-access-jtp9f\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:16:46.837744 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.837708    2562 generic.go:356] "Generic (PLEG): container finished" podID="290b4a4f-433f-43a0-9026-cc9d100e47af" containerID="02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e" exitCode=0
Sep 09 03:16:46.838235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.837754    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fplwj"
Sep 09 03:16:46.838235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.837792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fplwj" event={"ID":"290b4a4f-433f-43a0-9026-cc9d100e47af","Type":"ContainerDied","Data":"02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e"}
Sep 09 03:16:46.838235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.837838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fplwj" event={"ID":"290b4a4f-433f-43a0-9026-cc9d100e47af","Type":"ContainerDied","Data":"dec4e23ce107cd518df4c8b1ae5efb9edeeccfff3234ff26f4900b595f983111"}
Sep 09 03:16:46.838235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.837866    2562 scope.go:117] "RemoveContainer" containerID="02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e"
Sep 09 03:16:46.847212 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.847196    2562 scope.go:117] "RemoveContainer" containerID="02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e"
Sep 09 03:16:46.847494 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:16:46.847476    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e\": container with ID starting with 02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e not found: ID does not exist" containerID="02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e"
Sep 09 03:16:46.847569 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.847502    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e"} err="failed to get container status \"02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e\": rpc error: code = NotFound desc = could not find container \"02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e\": container with ID starting with 02e4106dcd336b88c30b340558261ecc6dae84e57ce6330b9a232c79095dee7e not found: ID does not exist"
Sep 09 03:16:46.857741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.857717    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fplwj"]
Sep 09 03:16:46.860879 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:46.860860    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fplwj"]
Sep 09 03:16:47.938487 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:47.938455    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290b4a4f-433f-43a0-9026-cc9d100e47af" path="/var/lib/kubelet/pods/290b4a4f-433f-43a0-9026-cc9d100e47af/volumes"
Sep 09 03:16:50.869759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:50.869723    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:50.979335 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:50.979286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:51.077557 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:51.077504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:51.251653 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:51.251558    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:53.950008 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:53.949970    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:54.049490 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:16:54.049458    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:00.302934 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:00.302883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:25.547738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:25.547697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:25.717135 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:25.717106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:26.957764 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:26.957730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:27.057843 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.057804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:27.162790 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.162749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:27.258733 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.258644    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:27.264751 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.264723    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"]
Sep 09 03:17:27.265121 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.265108    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290b4a4f-433f-43a0-9026-cc9d100e47af" containerName="authorino"
Sep 09 03:17:27.265176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.265123    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="290b4a4f-433f-43a0-9026-cc9d100e47af" containerName="authorino"
Sep 09 03:17:27.265215 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.265177    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="290b4a4f-433f-43a0-9026-cc9d100e47af" containerName="authorino"
Sep 09 03:17:27.273949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.273910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.276329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.276287    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6zwh9\""
Sep 09 03:17:27.276864 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.276837    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 03:17:27.277004 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.276963    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 03:17:27.278870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.278842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"]
Sep 09 03:17:27.372756 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.372726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-maas-api-tls\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.372957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.372836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-metrics-certs\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.372957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.372857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlfcc\" (UniqueName: \"kubernetes.io/projected/d6873e2b-79ae-4648-b8fc-dd569f4bc207-kube-api-access-hlfcc\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.372957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.372920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6873e2b-79ae-4648-b8fc-dd569f4bc207-openshift-service-ca\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.474042 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.474002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-maas-api-tls\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.474259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.474107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-metrics-certs\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.474259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.474206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlfcc\" (UniqueName: \"kubernetes.io/projected/d6873e2b-79ae-4648-b8fc-dd569f4bc207-kube-api-access-hlfcc\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.474259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.474253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6873e2b-79ae-4648-b8fc-dd569f4bc207-openshift-service-ca\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.474959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.474939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6873e2b-79ae-4648-b8fc-dd569f4bc207-openshift-service-ca\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.476823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.476800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-metrics-certs\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.476930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.476911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-maas-api-tls\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.482846 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.482817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlfcc\" (UniqueName: \"kubernetes.io/projected/d6873e2b-79ae-4648-b8fc-dd569f4bc207-kube-api-access-hlfcc\") pod \"maas-api-65bcb85cfb-jd2s8\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.526048 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.525950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:27.586667 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.586633    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:27.626734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.626689    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:27.863654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:27.863626    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"]
Sep 09 03:17:27.867156 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:17:27.867111    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6873e2b_79ae_4648_b8fc_dd569f4bc207.slice/crio-24c25410309aea136897c279a348c7bacbc91f18da23586a05c30e773da8d346 WatchSource:0}: Error finding container 24c25410309aea136897c279a348c7bacbc91f18da23586a05c30e773da8d346: Status 404 returned error can't find the container with id 24c25410309aea136897c279a348c7bacbc91f18da23586a05c30e773da8d346
Sep 09 03:17:28.073324 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:28.073281    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8" event={"ID":"d6873e2b-79ae-4648-b8fc-dd569f4bc207","Type":"ContainerStarted","Data":"24c25410309aea136897c279a348c7bacbc91f18da23586a05c30e773da8d346"}
Sep 09 03:17:30.090093 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:30.090002    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8" event={"ID":"d6873e2b-79ae-4648-b8fc-dd569f4bc207","Type":"ContainerStarted","Data":"072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d"}
Sep 09 03:17:30.090463 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:30.090127    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:30.106430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:30.106360    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8" podStartSLOduration=1.1549476140000001 podStartE2EDuration="3.106341881s" totalImagesPullingTime="1.951394267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:30.105154786 +0000 UTC m=+698.667481401" watchObservedRunningTime="2026-09-09 03:17:30.106341881 +0000 UTC m=+698.668668426"
Sep 09 03:17:36.102149 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:36.102119    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:17:37.522964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:37.522900    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:37.615119 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:37.615023    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:38.137094 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:38.137060    2562 generic.go:356] "Generic (PLEG): container finished" podID="9142fd43-df2e-4dc5-b146-029f01d15c7a" containerID="a0cad858001df230fbc4bddf99e31c638ef33e9996c1fc381a097a99168ec477" exitCode=0
Sep 09 03:17:38.137278 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:38.137107    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" event={"ID":"9142fd43-df2e-4dc5-b146-029f01d15c7a","Type":"ContainerDied","Data":"a0cad858001df230fbc4bddf99e31c638ef33e9996c1fc381a097a99168ec477"}
Sep 09 03:17:38.137501 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:38.137486    2562 scope.go:117] "RemoveContainer" containerID="a0cad858001df230fbc4bddf99e31c638ef33e9996c1fc381a097a99168ec477"
Sep 09 03:17:38.138682 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:38.138649    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:17:38.138760 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:38.138727    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:17:38.138801 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:38.138770    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:17:39.177146 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:39.177103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" event={"ID":"9142fd43-df2e-4dc5-b146-029f01d15c7a","Type":"ContainerStarted","Data":"7a030ee9b2c3fa54d8adb526d9a3ff8bd2b06334bc7cd2bbb4d4d166c04c36b2"}
Sep 09 03:17:39.179674 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:39.179651    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:17:39.179848 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:39.179822    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zvtkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 09 03:17:39.179985 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:39.179883    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" podUID="9142fd43-df2e-4dc5-b146-029f01d15c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 09 03:17:40.179463 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:40.179425    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zvtkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 09 03:17:40.179887 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:40.179496    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" podUID="9142fd43-df2e-4dc5-b146-029f01d15c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 09 03:17:41.179820 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:41.179784    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zvtkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 09 03:17:41.180211 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:41.179857    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" podUID="9142fd43-df2e-4dc5-b146-029f01d15c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 09 03:17:42.179293 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:42.179252    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zvtkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 09 03:17:42.179517 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:42.179326    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" podUID="9142fd43-df2e-4dc5-b146-029f01d15c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 09 03:17:43.179703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:43.179607    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zvtkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 09 03:17:43.179703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:43.179668    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" podUID="9142fd43-df2e-4dc5-b146-029f01d15c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 09 03:17:44.180025 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:44.179986    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zvtkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 09 03:17:44.180581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:44.180095    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx" podUID="9142fd43-df2e-4dc5-b146-029f01d15c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 09 03:17:45.183171 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:45.183137    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:17:45.183683 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:45.183653    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:17:45.184449 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:45.184429    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zvtkx"
Sep 09 03:17:49.837364 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:49.837325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:54.998116 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:54.998075    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"]
Sep 09 03:17:55.001271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.001242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.003705 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.003678    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 03:17:55.004383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.004365    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 03:17:55.004524 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.004367    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 03:17:55.004729 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.004706    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vxktl\""
Sep 09 03:17:55.010415 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.010387    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"]
Sep 09 03:17:55.028046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.028003    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:55.130259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.130220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.130259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.130263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d91c90-1905-4376-918a-03a200464429-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.130511 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.130296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9sj\" (UniqueName: \"kubernetes.io/projected/14d91c90-1905-4376-918a-03a200464429-kube-api-access-9b9sj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.130511 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.130369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.130511 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.130423    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.130511 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.130503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.231929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.231887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.231929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.231931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d91c90-1905-4376-918a-03a200464429-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.232213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.231956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9sj\" (UniqueName: \"kubernetes.io/projected/14d91c90-1905-4376-918a-03a200464429-kube-api-access-9b9sj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.232213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.231976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.232213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.231992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.232213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.232047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.232433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.232393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.232483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.232450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.232517 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.232484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.234603 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.234580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d91c90-1905-4376-918a-03a200464429-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.234736 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.234675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d91c90-1905-4376-918a-03a200464429-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.239961 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.239934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9sj\" (UniqueName: \"kubernetes.io/projected/14d91c90-1905-4376-918a-03a200464429-kube-api-access-9b9sj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7x6lw\" (UID: \"14d91c90-1905-4376-918a-03a200464429\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.316957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.316915    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:17:55.591854 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.591826    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"]
Sep 09 03:17:55.593869 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:17:55.593828    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d91c90_1905_4376_918a_03a200464429.slice/crio-eba527e68ed12943304ff860b9ef78405b5d99e1dea146fa6036ba2788522eaa WatchSource:0}: Error finding container eba527e68ed12943304ff860b9ef78405b5d99e1dea146fa6036ba2788522eaa: Status 404 returned error can't find the container with id eba527e68ed12943304ff860b9ef78405b5d99e1dea146fa6036ba2788522eaa
Sep 09 03:17:55.595338 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:55.595317    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:17:56.294344 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:56.294282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw" event={"ID":"14d91c90-1905-4376-918a-03a200464429","Type":"ContainerStarted","Data":"eba527e68ed12943304ff860b9ef78405b5d99e1dea146fa6036ba2788522eaa"}
Sep 09 03:17:57.301459 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:57.301423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw" event={"ID":"14d91c90-1905-4376-918a-03a200464429","Type":"ContainerStarted","Data":"5ee12e8605bd06700f2ec011579b57f41a037510bdee135cb35097639b341d87"}
Sep 09 03:17:57.319155 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:57.319097    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw" podStartSLOduration=2.2163906349999998 podStartE2EDuration="3.319078193s" totalImagesPullingTime="1.102687558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:57.31618079 +0000 UTC m=+725.878507349" watchObservedRunningTime="2026-09-09 03:17:57.319078193 +0000 UTC m=+725.881404737"
Sep 09 03:17:58.086799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:17:58.086761    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:04.291581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:04.291521    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:05.318045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:05.318003    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:18:05.318453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:05.318084    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:18:05.330893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:05.330865    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:18:05.362433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:05.362401    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7x6lw"
Sep 09 03:18:20.229150 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:20.229104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:35.312239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:35.312176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:41.596010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:41.595963    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:45.196433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.196382    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"]
Sep 09 03:18:45.199553 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.199525    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.201706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.201681    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 03:18:45.211337 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.211312    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"]
Sep 09 03:18:45.228916 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.228888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:45.277864 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.277826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746a5f45-210b-4dde-87af-15ffd135ac60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.277864 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.277870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.278129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.277924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tz7p\" (UniqueName: \"kubernetes.io/projected/746a5f45-210b-4dde-87af-15ffd135ac60-kube-api-access-7tz7p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.278129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.277956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.278129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.277990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.278129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.278076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379175 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746a5f45-210b-4dde-87af-15ffd135ac60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tz7p\" (UniqueName: \"kubernetes.io/projected/746a5f45-210b-4dde-87af-15ffd135ac60-kube-api-access-7tz7p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379665 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379763 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.379763 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.379714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.381803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.381780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746a5f45-210b-4dde-87af-15ffd135ac60-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.381990 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.381970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746a5f45-210b-4dde-87af-15ffd135ac60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.387991 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.387966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tz7p\" (UniqueName: \"kubernetes.io/projected/746a5f45-210b-4dde-87af-15ffd135ac60-kube-api-access-7tz7p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rpv6j\" (UID: \"746a5f45-210b-4dde-87af-15ffd135ac60\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.510802 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.510696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:45.785555 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:45.785507    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"]
Sep 09 03:18:45.787102 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:18:45.787067    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746a5f45_210b_4dde_87af_15ffd135ac60.slice/crio-60e2119fa3ed747539e98883af4712132ad726b31cbba182375c798da4d1e079 WatchSource:0}: Error finding container 60e2119fa3ed747539e98883af4712132ad726b31cbba182375c798da4d1e079: Status 404 returned error can't find the container with id 60e2119fa3ed747539e98883af4712132ad726b31cbba182375c798da4d1e079
Sep 09 03:18:46.594777 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:46.594735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j" event={"ID":"746a5f45-210b-4dde-87af-15ffd135ac60","Type":"ContainerStarted","Data":"aee47f17cb12a699ecfaebc623c931a095e7ba278c44a7b4b5172f017b8f4d91"}
Sep 09 03:18:46.595245 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:46.594790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j" event={"ID":"746a5f45-210b-4dde-87af-15ffd135ac60","Type":"ContainerStarted","Data":"60e2119fa3ed747539e98883af4712132ad726b31cbba182375c798da4d1e079"}
Sep 09 03:18:46.611908 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:46.611854    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j" podStartSLOduration=1.375657962 podStartE2EDuration="1.611837997s" totalImagesPullingTime="236.180035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:46.61045708 +0000 UTC m=+775.172783625" watchObservedRunningTime="2026-09-09 03:18:46.611837997 +0000 UTC m=+775.174164542"
Sep 09 03:18:55.511164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:55.511119    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:55.511713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:55.511199    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:55.522988 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:55.522950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:55.524649 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:55.524622    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:18:55.666213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:18:55.666185    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rpv6j"
Sep 09 03:19:08.857372 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:08.857333    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:08.959131 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:08.959093    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:39.721992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:39.721886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:59.589437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:59.589381    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cd969b7ff-cdfbs"]
Sep 09 03:19:59.590167 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:59.589971    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs" podUID="00cfb66b-52be-4540-8f43-777077c2cb08" containerName="authorino" containerID="cri-o://6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e" gracePeriod=30
Sep 09 03:19:59.864010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:59.863981    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:19:59.969757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:59.969725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98vtk\" (UniqueName: \"kubernetes.io/projected/00cfb66b-52be-4540-8f43-777077c2cb08-kube-api-access-98vtk\") pod \"00cfb66b-52be-4540-8f43-777077c2cb08\" (UID: \"00cfb66b-52be-4540-8f43-777077c2cb08\") "
Sep 09 03:19:59.969957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:59.969786    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00cfb66b-52be-4540-8f43-777077c2cb08-tls-cert\") pod \"00cfb66b-52be-4540-8f43-777077c2cb08\" (UID: \"00cfb66b-52be-4540-8f43-777077c2cb08\") "
Sep 09 03:19:59.972283 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:59.972245    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cfb66b-52be-4540-8f43-777077c2cb08-kube-api-access-98vtk" (OuterVolumeSpecName: "kube-api-access-98vtk") pod "00cfb66b-52be-4540-8f43-777077c2cb08" (UID: "00cfb66b-52be-4540-8f43-777077c2cb08"). InnerVolumeSpecName "kube-api-access-98vtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:19:59.981600 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:19:59.981569    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00cfb66b-52be-4540-8f43-777077c2cb08-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "00cfb66b-52be-4540-8f43-777077c2cb08" (UID: "00cfb66b-52be-4540-8f43-777077c2cb08"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:20:00.041610 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.041568    2562 generic.go:356] "Generic (PLEG): container finished" podID="00cfb66b-52be-4540-8f43-777077c2cb08" containerID="6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e" exitCode=0
Sep 09 03:20:00.041610 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.041613    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs"
Sep 09 03:20:00.041859 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.041692    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs" event={"ID":"00cfb66b-52be-4540-8f43-777077c2cb08","Type":"ContainerDied","Data":"6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e"}
Sep 09 03:20:00.041859 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.041726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd969b7ff-cdfbs" event={"ID":"00cfb66b-52be-4540-8f43-777077c2cb08","Type":"ContainerDied","Data":"31d1b579ef3a1e460eb0be7fd4f15d6ea0d6a878a538d13de71f63dc069f9897"}
Sep 09 03:20:00.041859 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.041746    2562 scope.go:117] "RemoveContainer" containerID="6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e"
Sep 09 03:20:00.055522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.055495    2562 scope.go:117] "RemoveContainer" containerID="6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e"
Sep 09 03:20:00.055986 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:20:00.055926    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e\": container with ID starting with 6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e not found: ID does not exist" containerID="6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e"
Sep 09 03:20:00.056095 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.056001    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e"} err="failed to get container status \"6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e\": rpc error: code = NotFound desc = could not find container \"6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e\": container with ID starting with 6eeb2a45facc5ce201c3a2448cede0e73b854b716a61d4615fa69a87b016ca1e not found: ID does not exist"
Sep 09 03:20:00.068778 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.068745    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cd969b7ff-cdfbs"]
Sep 09 03:20:00.070734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.070702    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98vtk\" (UniqueName: \"kubernetes.io/projected/00cfb66b-52be-4540-8f43-777077c2cb08-kube-api-access-98vtk\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:20:00.070734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.070735    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00cfb66b-52be-4540-8f43-777077c2cb08-tls-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:20:00.073167 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.073137    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cd969b7ff-cdfbs"]
Sep 09 03:20:00.087334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.087292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:00.190068 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:00.189961    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:01.228179 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:01.228143    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:01.370054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:01.370016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:01.938995 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:01.938961    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00cfb66b-52be-4540-8f43-777077c2cb08" path="/var/lib/kubelet/pods/00cfb66b-52be-4540-8f43-777077c2cb08/volumes"
Sep 09 03:20:16.436929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.436891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:16.730914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.730821    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"]
Sep 09 03:20:16.731417 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.731401    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cfb66b-52be-4540-8f43-777077c2cb08" containerName="authorino"
Sep 09 03:20:16.731479 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.731422    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cfb66b-52be-4540-8f43-777077c2cb08" containerName="authorino"
Sep 09 03:20:16.731572 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.731531    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00cfb66b-52be-4540-8f43-777077c2cb08" containerName="authorino"
Sep 09 03:20:16.735026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.734999    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.737789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.737766    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b27dde-openshift-default-dockercfg-2nvzs\""
Sep 09 03:20:16.766920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.766888    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"]
Sep 09 03:20:16.826777 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.826742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-token\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.826979 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.826788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-certs\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.826979 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.826862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw2wn\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-kube-api-access-mw2wn\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.826979 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.826907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-data\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.826979 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.826940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-podinfo\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.827121 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.827056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-envoy\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.827121 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.827100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-socket\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.827191 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.827174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-credential-socket\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.827273 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.827209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c71dda-27bd-430a-8e0b-28a3f277f372-istiod-ca-cert\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.928857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.928817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-token\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.928868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-certs\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.928902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw2wn\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-kube-api-access-mw2wn\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.928928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-data\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.928959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-podinfo\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-envoy\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-socket\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-credential-socket\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c71dda-27bd-430a-8e0b-28a3f277f372-istiod-ca-cert\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929623 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-data\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929694 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-credential-socket\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929694 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-certs\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.929799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-socket\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.930009 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.929991    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c71dda-27bd-430a-8e0b-28a3f277f372-istiod-ca-cert\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.931722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.931699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-envoy\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.932117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.932100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-podinfo\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.937023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.936997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-token\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:16.937441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:16.937416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw2wn\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-kube-api-access-mw2wn\") pod \"e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") " pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:17.048750 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:17.048719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:17.330487 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:17.330449    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"]
Sep 09 03:20:17.331862 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:20:17.331837    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c71dda_27bd_430a_8e0b_28a3f277f372.slice/crio-8831d6ecfd4691acba7ee29ad2bbd9693f99bc338e7166bd0aa9949846a063f1 WatchSource:0}: Error finding container 8831d6ecfd4691acba7ee29ad2bbd9693f99bc338e7166bd0aa9949846a063f1: Status 404 returned error can't find the container with id 8831d6ecfd4691acba7ee29ad2bbd9693f99bc338e7166bd0aa9949846a063f1
Sep 09 03:20:17.334501 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:17.334468    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:17.334634 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:17.334553    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:17.334634 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:17.334589    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:18.146424 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.146382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" event={"ID":"84c71dda-27bd-430a-8e0b-28a3f277f372","Type":"ContainerStarted","Data":"e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e"}
Sep 09 03:20:18.146941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.146459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" event={"ID":"84c71dda-27bd-430a-8e0b-28a3f277f372","Type":"ContainerStarted","Data":"8831d6ecfd4691acba7ee29ad2bbd9693f99bc338e7166bd0aa9949846a063f1"}
Sep 09 03:20:18.172318 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.172267    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" podStartSLOduration=2.172248995 podStartE2EDuration="2.172248995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:18.170927999 +0000 UTC m=+866.733254545" watchObservedRunningTime="2026-09-09 03:20:18.172248995 +0000 UTC m=+866.734575608"
Sep 09 03:20:18.304505 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.304467    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:18.308051 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.308029    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:18.311809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.311787    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-49nx9\""
Sep 09 03:20:18.314827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.314799    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:18.363738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.363708    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:18.443783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.443670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-582dx\" (UniqueName: \"kubernetes.io/projected/98c7aa20-06a4-4e80-abbc-50472fbaf763-kube-api-access-582dx\") pod \"test-curl-7613-139818062651184\" (UID: \"98c7aa20-06a4-4e80-abbc-50472fbaf763\") " pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:18.544404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.544356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-582dx\" (UniqueName: \"kubernetes.io/projected/98c7aa20-06a4-4e80-abbc-50472fbaf763-kube-api-access-582dx\") pod \"test-curl-7613-139818062651184\" (UID: \"98c7aa20-06a4-4e80-abbc-50472fbaf763\") " pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:18.552623 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.552589    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-582dx\" (UniqueName: \"kubernetes.io/projected/98c7aa20-06a4-4e80-abbc-50472fbaf763-kube-api-access-582dx\") pod \"test-curl-7613-139818062651184\" (UID: \"98c7aa20-06a4-4e80-abbc-50472fbaf763\") " pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:18.622224 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.622188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:18.893353 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:18.893323    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:18.894762 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:20:18.894735    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98c7aa20_06a4_4e80_abbc_50472fbaf763.slice/crio-068c6f5a637a203120608cf0e797bbe13c0322f68c09f0fa39018177e4a8dc2a WatchSource:0}: Error finding container 068c6f5a637a203120608cf0e797bbe13c0322f68c09f0fa39018177e4a8dc2a: Status 404 returned error can't find the container with id 068c6f5a637a203120608cf0e797bbe13c0322f68c09f0fa39018177e4a8dc2a
Sep 09 03:20:19.049454 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:19.049420    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:19.056168 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:19.056140    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:19.154155 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:19.154064    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139818062651184" event={"ID":"98c7aa20-06a4-4e80-abbc-50472fbaf763","Type":"ContainerStarted","Data":"068c6f5a637a203120608cf0e797bbe13c0322f68c09f0fa39018177e4a8dc2a"}
Sep 09 03:20:19.154592 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:19.154521    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:19.155636 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:19.155611    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:20.067023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:20.066982    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:20.159436 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:20.159395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:20.162418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:20.162393    2562 generic.go:356] "Generic (PLEG): container finished" podID="98c7aa20-06a4-4e80-abbc-50472fbaf763" containerID="2496bde1e6974446259779888115080776b51ebba59084add9842f7f8fbfb585" exitCode=0
Sep 09 03:20:20.162576 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:20.162485    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139818062651184" event={"ID":"98c7aa20-06a4-4e80-abbc-50472fbaf763","Type":"ContainerDied","Data":"2496bde1e6974446259779888115080776b51ebba59084add9842f7f8fbfb585"}
Sep 09 03:20:21.231516 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.231480    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"]
Sep 09 03:20:21.236975 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.236944    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.253597 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.250428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"]
Sep 09 03:20:21.253597 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.250596    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b27dde-serving-cert\""
Sep 09 03:20:21.274823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.274667    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:21.347640 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.347613    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:21.370166 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.370131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4d037d-0f61-4ace-8667-230d796a2a1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.370166 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.370168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.370430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.370191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qj7\" (UniqueName: \"kubernetes.io/projected/5b4d037d-0f61-4ace-8667-230d796a2a1d-kube-api-access-g2qj7\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.370430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.370272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-metrics-certs\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.371092 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.371063    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:21.378656 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.378619    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7613-139818062651184_98c7aa20-06a4-4e80-abbc-50472fbaf763/test-curl-7613-139818062651184/0.log"
Sep 09 03:20:21.400970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.400938    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:21.408812 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.408779    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:21.471622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.470887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-582dx\" (UniqueName: \"kubernetes.io/projected/98c7aa20-06a4-4e80-abbc-50472fbaf763-kube-api-access-582dx\") pod \"98c7aa20-06a4-4e80-abbc-50472fbaf763\" (UID: \"98c7aa20-06a4-4e80-abbc-50472fbaf763\") "
Sep 09 03:20:21.471622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.471178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4d037d-0f61-4ace-8667-230d796a2a1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.471622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.471213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.471622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.471244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qj7\" (UniqueName: \"kubernetes.io/projected/5b4d037d-0f61-4ace-8667-230d796a2a1d-kube-api-access-g2qj7\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.471622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.471307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-metrics-certs\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.472090 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.471992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4d037d-0f61-4ace-8667-230d796a2a1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.474477 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.473371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:21.474753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.474727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c7aa20-06a4-4e80-abbc-50472fbaf763-kube-api-access-582dx" (OuterVolumeSpecName: "kube-api-access-582dx") pod "98c7aa20-06a4-4e80-abbc-50472fbaf763" (UID: "98c7aa20-06a4-4e80-abbc-50472fbaf763"). InnerVolumeSpecName "kube-api-access-582dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:20:21.474869 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.474772    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.475042 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.475023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-metrics-certs\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.481430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.481408    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qj7\" (UniqueName: \"kubernetes.io/projected/5b4d037d-0f61-4ace-8667-230d796a2a1d-kube-api-access-g2qj7\") pod \"maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.524343 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.524309    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:21.524768 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.524753    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c7aa20-06a4-4e80-abbc-50472fbaf763" containerName="test-curl-7613-139818062651184"
Sep 09 03:20:21.524829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.524770    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c7aa20-06a4-4e80-abbc-50472fbaf763" containerName="test-curl-7613-139818062651184"
Sep 09 03:20:21.524867 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.524841    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98c7aa20-06a4-4e80-abbc-50472fbaf763" containerName="test-curl-7613-139818062651184"
Sep 09 03:20:21.528143 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.528124    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:21.532647 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.532559    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:21.564308 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.564269    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:21.572085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.572050    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-582dx\" (UniqueName: \"kubernetes.io/projected/98c7aa20-06a4-4e80-abbc-50472fbaf763-kube-api-access-582dx\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:20:21.673560 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.673459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdrq\" (UniqueName: \"kubernetes.io/projected/2936ad7a-4df1-4095-a2bf-1854a8a75f6f-kube-api-access-drdrq\") pod \"test-curl-7613-139818062651184\" (UID: \"2936ad7a-4df1-4095-a2bf-1854a8a75f6f\") " pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:21.774948 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.774842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdrq\" (UniqueName: \"kubernetes.io/projected/2936ad7a-4df1-4095-a2bf-1854a8a75f6f-kube-api-access-drdrq\") pod \"test-curl-7613-139818062651184\" (UID: \"2936ad7a-4df1-4095-a2bf-1854a8a75f6f\") " pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:21.784176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.784141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdrq\" (UniqueName: \"kubernetes.io/projected/2936ad7a-4df1-4095-a2bf-1854a8a75f6f-kube-api-access-drdrq\") pod \"test-curl-7613-139818062651184\" (UID: \"2936ad7a-4df1-4095-a2bf-1854a8a75f6f\") " pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:21.801977 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.801945    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:21.844964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.844699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:21.856097 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.856049    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"]
Sep 09 03:20:21.862671 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:20:21.862627    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4d037d_0f61_4ace_8667_230d796a2a1d.slice/crio-8aa924252427c1042658b88009fa7d1c558b0e7616f7478140d7ebdd3d360398 WatchSource:0}: Error finding container 8aa924252427c1042658b88009fa7d1c558b0e7616f7478140d7ebdd3d360398: Status 404 returned error can't find the container with id 8aa924252427c1042658b88009fa7d1c558b0e7616f7478140d7ebdd3d360398
Sep 09 03:20:21.903823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.903731    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:21.962248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:21.962215    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98c7aa20-06a4-4e80-abbc-50472fbaf763" path="/var/lib/kubelet/pods/98c7aa20-06a4-4e80-abbc-50472fbaf763/volumes"
Sep 09 03:20:22.147462 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:22.147430    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:22.149619 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:20:22.149590    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2936ad7a_4df1_4095_a2bf_1854a8a75f6f.slice/crio-72788c416713b4202581fa9b38471e91474d9006318851809870656ff2a819e0 WatchSource:0}: Error finding container 72788c416713b4202581fa9b38471e91474d9006318851809870656ff2a819e0: Status 404 returned error can't find the container with id 72788c416713b4202581fa9b38471e91474d9006318851809870656ff2a819e0
Sep 09 03:20:22.176257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:22.176226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg" event={"ID":"5b4d037d-0f61-4ace-8667-230d796a2a1d","Type":"ContainerStarted","Data":"8aa924252427c1042658b88009fa7d1c558b0e7616f7478140d7ebdd3d360398"}
Sep 09 03:20:22.177472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:22.177450    2562 scope.go:117] "RemoveContainer" containerID="2496bde1e6974446259779888115080776b51ebba59084add9842f7f8fbfb585"
Sep 09 03:20:22.177472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:22.177469    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:22.178857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:22.178835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139818062651184" event={"ID":"2936ad7a-4df1-4095-a2bf-1854a8a75f6f","Type":"ContainerStarted","Data":"72788c416713b4202581fa9b38471e91474d9006318851809870656ff2a819e0"}
Sep 09 03:20:23.186728 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:23.186688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg" event={"ID":"5b4d037d-0f61-4ace-8667-230d796a2a1d","Type":"ContainerStarted","Data":"a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d"}
Sep 09 03:20:23.187207 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:23.186885    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:23.189273 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:23.189246    2562 generic.go:356] "Generic (PLEG): container finished" podID="2936ad7a-4df1-4095-a2bf-1854a8a75f6f" containerID="6bea4595886be94868cd378b8d93ad98fa39dd00262a28cd81c7e10517a7819d" exitCode=0
Sep 09 03:20:23.189404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:23.189338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139818062651184" event={"ID":"2936ad7a-4df1-4095-a2bf-1854a8a75f6f","Type":"ContainerDied","Data":"6bea4595886be94868cd378b8d93ad98fa39dd00262a28cd81c7e10517a7819d"}
Sep 09 03:20:23.205022 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:23.204969    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg" podStartSLOduration=1.689758182 podStartE2EDuration="2.204952843s" totalImagesPullingTime="515.194661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:23.203593683 +0000 UTC m=+871.765920224" watchObservedRunningTime="2026-09-09 03:20:23.204952843 +0000 UTC m=+871.767279382"
Sep 09 03:20:24.254111 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.254081    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:24.342792 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.342762    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:24.353614 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.353585    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:24.362261 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.362233    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7613-139818062651184_2936ad7a-4df1-4095-a2bf-1854a8a75f6f/test-curl-7613-139818062651184/0.log"
Sep 09 03:20:24.383902 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.383877    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:24.388649 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.388627    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7613-139818062651184"]
Sep 09 03:20:24.473758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.473665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:24.498488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.498456    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drdrq\" (UniqueName: \"kubernetes.io/projected/2936ad7a-4df1-4095-a2bf-1854a8a75f6f-kube-api-access-drdrq\") pod \"2936ad7a-4df1-4095-a2bf-1854a8a75f6f\" (UID: \"2936ad7a-4df1-4095-a2bf-1854a8a75f6f\") "
Sep 09 03:20:24.500803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.500776    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2936ad7a-4df1-4095-a2bf-1854a8a75f6f-kube-api-access-drdrq" (OuterVolumeSpecName: "kube-api-access-drdrq") pod "2936ad7a-4df1-4095-a2bf-1854a8a75f6f" (UID: "2936ad7a-4df1-4095-a2bf-1854a8a75f6f"). InnerVolumeSpecName "kube-api-access-drdrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:20:24.599464 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.599426    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdrq\" (UniqueName: \"kubernetes.io/projected/2936ad7a-4df1-4095-a2bf-1854a8a75f6f-kube-api-access-drdrq\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:20:24.780934 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.780861    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"]
Sep 09 03:20:24.781350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.781322    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2936ad7a-4df1-4095-a2bf-1854a8a75f6f" containerName="test-curl-7613-139818062651184"
Sep 09 03:20:24.781350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.781350    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2936ad7a-4df1-4095-a2bf-1854a8a75f6f" containerName="test-curl-7613-139818062651184"
Sep 09 03:20:24.781574 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.781487    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2936ad7a-4df1-4095-a2bf-1854a8a75f6f" containerName="test-curl-7613-139818062651184"
Sep 09 03:20:24.784936 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.784916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.787159 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.787135    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5ecf63f7-openshift-default-dockercfg-9ghvl\""
Sep 09 03:20:24.796946 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.796924    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"]
Sep 09 03:20:24.804875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.804847    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:24.871771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.871729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:24.902331 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-data\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.902484 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc284174-f17e-4f26-9ba7-723976502a7e-istio-podinfo\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.902484 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc284174-f17e-4f26-9ba7-723976502a7e-istiod-ca-cert\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.902484 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902397    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-envoy\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.902484 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-istio-token\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.902484 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dthkq\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-kube-api-access-dthkq\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.902670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902563    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-certs\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.902670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-socket\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.902670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.902653    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-credential-socket\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:24.972075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:24.972043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:25.003335 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-envoy\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003335 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-istio-token\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003530 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dthkq\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-kube-api-access-dthkq\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003617 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-certs\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003617 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-socket\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003696 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-credential-socket\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003770 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-data\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc284174-f17e-4f26-9ba7-723976502a7e-istio-podinfo\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003876 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003837    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc284174-f17e-4f26-9ba7-723976502a7e-istiod-ca-cert\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-certs\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.003935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-socket\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.004042 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.003974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-credential-socket\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.004180 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.004158    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-data\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.004806 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.004780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc284174-f17e-4f26-9ba7-723976502a7e-istiod-ca-cert\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.005925 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.005904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-envoy\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.006288 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.006273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc284174-f17e-4f26-9ba7-723976502a7e-istio-podinfo\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.011359 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.011337    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-istio-token\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.011851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.011832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dthkq\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-kube-api-access-dthkq\") pod \"e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") " pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.097788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.097758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:25.171573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.171147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:25.215408 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.215377    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:25.215735 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.215391    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72788c416713b4202581fa9b38471e91474d9006318851809870656ff2a819e0"
Sep 09 03:20:25.228783 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:20:25.228742    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139818062651184\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="2936ad7a-4df1-4095-a2bf-1854a8a75f6f" pod="opendatahub/test-curl-7613-139818062651184"
Sep 09 03:20:25.398071 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.398039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"]
Sep 09 03:20:25.399279 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:20:25.399245    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc284174_f17e_4f26_9ba7_723976502a7e.slice/crio-a3be5d9e485a8f84e76f3742214a94be98944ae7e88d6cd9d85898ed0667b148 WatchSource:0}: Error finding container a3be5d9e485a8f84e76f3742214a94be98944ae7e88d6cd9d85898ed0667b148: Status 404 returned error can't find the container with id a3be5d9e485a8f84e76f3742214a94be98944ae7e88d6cd9d85898ed0667b148
Sep 09 03:20:25.402050 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.402014    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:25.402138 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.402101    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:25.402138 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.402131    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:25.453102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.453073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:25.938975 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:25.938940    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2936ad7a-4df1-4095-a2bf-1854a8a75f6f" path="/var/lib/kubelet/pods/2936ad7a-4df1-4095-a2bf-1854a8a75f6f/volumes"
Sep 09 03:20:26.224117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:26.224032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" event={"ID":"cc284174-f17e-4f26-9ba7-723976502a7e","Type":"ContainerStarted","Data":"96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556"}
Sep 09 03:20:26.224117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:26.224095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" event={"ID":"cc284174-f17e-4f26-9ba7-723976502a7e","Type":"ContainerStarted","Data":"a3be5d9e485a8f84e76f3742214a94be98944ae7e88d6cd9d85898ed0667b148"}
Sep 09 03:20:26.247501 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:26.247450    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" podStartSLOduration=2.247388362 podStartE2EDuration="2.247388362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:26.246323138 +0000 UTC m=+874.808649706" watchObservedRunningTime="2026-09-09 03:20:26.247388362 +0000 UTC m=+874.809714908"
Sep 09 03:20:27.098896 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:27.098859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:27.104286 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:27.104261    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:27.229860 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:27.229821    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:27.230822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:27.230802    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:27.576737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:27.576660    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:27.676175 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:27.676138    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:28.696830 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:28.696778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:28.802064 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:28.801981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:29.036283 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.036252    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"]
Sep 09 03:20:29.040570 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.040516    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.049573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.047130    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5ecf63f7-serving-cert\""
Sep 09 03:20:29.057565 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.055255    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"]
Sep 09 03:20:29.097575 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.095928    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:29.163116 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.163068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19839c-9823-45b4-b2b7-116f930526b3-openshift-service-ca\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.163321 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.163124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-metrics-certs\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.163321 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.163182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcb4\" (UniqueName: \"kubernetes.io/projected/df19839c-9823-45b4-b2b7-116f930526b3-kube-api-access-btcb4\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.163321 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.163230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.185062 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.185026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:29.201289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.201263    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:20:29.265077 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.265005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19839c-9823-45b4-b2b7-116f930526b3-openshift-service-ca\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.265077 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.265057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-metrics-certs\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.265328 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.265116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcb4\" (UniqueName: \"kubernetes.io/projected/df19839c-9823-45b4-b2b7-116f930526b3-kube-api-access-btcb4\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.265328 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.265164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.265429 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:20:29.265361    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-serving-cert: secret "maas-api-e2e-ait-5ecf63f7-serving-cert" not found
Sep 09 03:20:29.265429 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:20:29.265424    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls podName:df19839c-9823-45b4-b2b7-116f930526b3 nodeName:}" failed. No retries permitted until 2026-09-09 03:20:29.765403143 +0000 UTC m=+878.327729684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls") pod "maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8" (UID: "df19839c-9823-45b4-b2b7-116f930526b3") : secret "maas-api-e2e-ait-5ecf63f7-serving-cert" not found
Sep 09 03:20:29.280566 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.266370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19839c-9823-45b4-b2b7-116f930526b3-openshift-service-ca\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.282437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.282409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-metrics-certs\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.287376 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.287304    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcb4\" (UniqueName: \"kubernetes.io/projected/df19839c-9823-45b4-b2b7-116f930526b3-kube-api-access-btcb4\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.666343 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.666252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:29.766722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.766686    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:29.772067 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:29.772036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:29.772633 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:20:29.772295    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-serving-cert: secret "maas-api-e2e-ait-5ecf63f7-serving-cert" not found
Sep 09 03:20:29.772749 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:20:29.772694    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls podName:df19839c-9823-45b4-b2b7-116f930526b3 nodeName:}" failed. No retries permitted until 2026-09-09 03:20:30.772670958 +0000 UTC m=+879.334997489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls") pod "maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8" (UID: "df19839c-9823-45b4-b2b7-116f930526b3") : secret "maas-api-e2e-ait-5ecf63f7-serving-cert" not found
Sep 09 03:20:30.104903 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.104858    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"]
Sep 09 03:20:30.109498 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.109478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.112183 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.112157    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-be414f-serving-cert\""
Sep 09 03:20:30.119515 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.119487    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"]
Sep 09 03:20:30.138893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.138855    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:30.182455 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.182422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6kk\" (UniqueName: \"kubernetes.io/projected/d3c77ffb-20ee-4c1f-a88b-856193ee8214-kube-api-access-ql6kk\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.182455 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.182461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-metrics-certs\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.182734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.182490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.182734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.182614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c77ffb-20ee-4c1f-a88b-856193ee8214-openshift-service-ca\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.238709 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.238671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:30.283257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.283219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6kk\" (UniqueName: \"kubernetes.io/projected/d3c77ffb-20ee-4c1f-a88b-856193ee8214-kube-api-access-ql6kk\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.283437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.283265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-metrics-certs\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.283480 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.283432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.283562 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.283522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c77ffb-20ee-4c1f-a88b-856193ee8214-openshift-service-ca\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.283624 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:20:30.283603    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-be414f-serving-cert: secret "maas-api-e2e-autores-be414f-serving-cert" not found
Sep 09 03:20:30.283694 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:20:30.283682    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls podName:d3c77ffb-20ee-4c1f-a88b-856193ee8214 nodeName:}" failed. No retries permitted until 2026-09-09 03:20:30.783661237 +0000 UTC m=+879.345987773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls") pod "maas-api-e2e-autores-be414f-8586747548-582x4" (UID: "d3c77ffb-20ee-4c1f-a88b-856193ee8214") : secret "maas-api-e2e-autores-be414f-serving-cert" not found
Sep 09 03:20:30.284171 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.284152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c77ffb-20ee-4c1f-a88b-856193ee8214-openshift-service-ca\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.286004 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.285982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-metrics-certs\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.292224 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.292199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6kk\" (UniqueName: \"kubernetes.io/projected/d3c77ffb-20ee-4c1f-a88b-856193ee8214-kube-api-access-ql6kk\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.339834 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.339791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:30.611400 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.611369    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:30.708619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.708575    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:30.786947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.786907    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:30.787445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.786974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.789975 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.789951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls\") pod \"maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:30.790097 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.789976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls\") pod \"maas-api-e2e-autores-be414f-8586747548-582x4\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:30.882465 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:30.882383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:31.025821 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:31.025783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:31.175850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:31.174391    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:31.175850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:31.174930    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"]
Sep 09 03:20:31.179418 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:20:31.179379    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf19839c_9823_45b4_b2b7_116f930526b3.slice/crio-043580ff81e6fcdb513c191f3262fb12712e486ef08a9204a1be8d65a880eeb0 WatchSource:0}: Error finding container 043580ff81e6fcdb513c191f3262fb12712e486ef08a9204a1be8d65a880eeb0: Status 404 returned error can't find the container with id 043580ff81e6fcdb513c191f3262fb12712e486ef08a9204a1be8d65a880eeb0
Sep 09 03:20:31.273578 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:31.273512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8" event={"ID":"df19839c-9823-45b4-b2b7-116f930526b3","Type":"ContainerStarted","Data":"043580ff81e6fcdb513c191f3262fb12712e486ef08a9204a1be8d65a880eeb0"}
Sep 09 03:20:31.318574 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:31.318526    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"]
Sep 09 03:20:31.319681 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:20:31.319655    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c77ffb_20ee_4c1f_a88b_856193ee8214.slice/crio-07896f77ea571a30de7e3f223717f895b74f261a6c688d183afe5f45913ab9f9 WatchSource:0}: Error finding container 07896f77ea571a30de7e3f223717f895b74f261a6c688d183afe5f45913ab9f9: Status 404 returned error can't find the container with id 07896f77ea571a30de7e3f223717f895b74f261a6c688d183afe5f45913ab9f9
Sep 09 03:20:31.905113 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:31.905079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:32.280344 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:32.280307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4" event={"ID":"d3c77ffb-20ee-4c1f-a88b-856193ee8214","Type":"ContainerStarted","Data":"097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594"}
Sep 09 03:20:32.280586 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:32.280372    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4" event={"ID":"d3c77ffb-20ee-4c1f-a88b-856193ee8214","Type":"ContainerStarted","Data":"07896f77ea571a30de7e3f223717f895b74f261a6c688d183afe5f45913ab9f9"}
Sep 09 03:20:32.280586 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:32.280526    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:32.281911 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:32.281893    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8" event={"ID":"df19839c-9823-45b4-b2b7-116f930526b3","Type":"ContainerStarted","Data":"c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39"}
Sep 09 03:20:32.281999 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:32.281982    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:32.296767 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:32.296726    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4" podStartSLOduration=1.757176112 podStartE2EDuration="2.29670386s" totalImagesPullingTime="539.527748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:32.295769753 +0000 UTC m=+880.858096297" watchObservedRunningTime="2026-09-09 03:20:32.29670386 +0000 UTC m=+880.859030405"
Sep 09 03:20:32.312196 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:32.312138    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8" podStartSLOduration=2.7760328100000002 podStartE2EDuration="3.312122184s" totalImagesPullingTime="536.089374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:32.311589103 +0000 UTC m=+880.873915648" watchObservedRunningTime="2026-09-09 03:20:32.312122184 +0000 UTC m=+880.874448730"
Sep 09 03:20:35.195834 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:35.195763    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 03:20:35.196301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:35.195884    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 03:20:35.343631 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:35.343557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:35.495979 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:35.495875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:35.591290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:35.591260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:35.692707 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:35.692668    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:37.315833 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:37.315802    2562 generic.go:356] "Generic (PLEG): container finished" podID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerID="e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e" exitCode=0
Sep 09 03:20:37.316274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:37.315885    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" event={"ID":"84c71dda-27bd-430a-8e0b-28a3f277f372","Type":"ContainerDied","Data":"e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e"}
Sep 09 03:20:37.316361 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:37.316347    2562 scope.go:117] "RemoveContainer" containerID="e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e"
Sep 09 03:20:37.317641 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:37.317601    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:37.317759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:37.317689    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:37.317759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:37.317724    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:38.298239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:38.298198    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:20:38.298488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:38.298464    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:20:38.325251 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:38.325212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" event={"ID":"84c71dda-27bd-430a-8e0b-28a3f277f372","Type":"ContainerStarted","Data":"13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66"}
Sep 09 03:20:38.395962 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:38.395926    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:39.049503 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:39.049460    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:39.049745 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:39.049623    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 03:20:39.049745 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:39.049675    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 03:20:39.962834 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:39.962787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:40.050223 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:40.050184    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 03:20:40.050670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:40.050633    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 03:20:41.050062 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:41.050020    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 03:20:41.050727 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:41.050691    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 03:20:42.049336 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:42.049291    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 03:20:42.049667 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:42.049632    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 03:20:42.272929 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:42.272885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:43.050112 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:43.050067    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 03:20:43.050333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:43.050148    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 03:20:43.957646 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:43.957605    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:44.057122 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:44.057030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:44.060316 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:44.060292    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:44.060779 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:44.060713    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:44.064000 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:44.063978    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:20:44.255873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:44.255782    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:44.986673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:44.986630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:45.297659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:45.297620    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:45.735311 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:45.735206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:46.047668 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:46.047513    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:47.398124 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:47.398085    2562 generic.go:356] "Generic (PLEG): container finished" podID="cc284174-f17e-4f26-9ba7-723976502a7e" containerID="96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556" exitCode=0
Sep 09 03:20:47.398620 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:47.398160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" event={"ID":"cc284174-f17e-4f26-9ba7-723976502a7e","Type":"ContainerDied","Data":"96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556"}
Sep 09 03:20:47.398678 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:47.398620    2562 scope.go:117] "RemoveContainer" containerID="96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556"
Sep 09 03:20:47.399726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:47.399692    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:47.399857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:47.399779    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:47.399857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:47.399828    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:20:48.407131 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:48.407089    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" event={"ID":"cc284174-f17e-4f26-9ba7-723976502a7e","Type":"ContainerStarted","Data":"21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994"}
Sep 09 03:20:48.750533 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:48.750440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:48.833142 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:48.833104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:49.098901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:49.098865    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:49.099152 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:49.099128    2562 patch_prober.go:28] interesting pod/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 09 03:20:49.099237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:49.099188    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 09 03:20:49.910904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:49.910862    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"]
Sep 09 03:20:49.913619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:49.913599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:49.923253 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:49.923224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-70784b-serving-cert\""
Sep 09 03:20:49.979643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:49.979595    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"]
Sep 09 03:20:50.005401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.005366    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:50.027334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.027304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crj4h\" (UniqueName: \"kubernetes.io/projected/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-kube-api-access-crj4h\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.027334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.027338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.027599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.027359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-maas-api-tls\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.027599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.027381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-metrics-certs\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.098615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.098576    2562 patch_prober.go:28] interesting pod/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 09 03:20:50.098805 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.098636    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 09 03:20:50.123649 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.123608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:50.128767 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.128741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crj4h\" (UniqueName: \"kubernetes.io/projected/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-kube-api-access-crj4h\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.128877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.128778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.128877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.128798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-maas-api-tls\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.128877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.128825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-metrics-certs\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.129493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.129473    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.131726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.131702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-metrics-certs\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.131850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.131754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-maas-api-tls\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.136878 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.136845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crj4h\" (UniqueName: \"kubernetes.io/projected/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-kube-api-access-crj4h\") pod \"maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.224061 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.223968    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:50.225898 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.225878    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:50.535789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.535753    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"]
Sep 09 03:20:50.536262 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:20:50.536227    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d5ed21_4e4a_4287_a01c_0a3bb93e3637.slice/crio-fe9dedce34d446d8b021d2e8bc3ebd64203adc876fc2913ba2d85d300cbc09cc WatchSource:0}: Error finding container fe9dedce34d446d8b021d2e8bc3ebd64203adc876fc2913ba2d85d300cbc09cc: Status 404 returned error can't find the container with id fe9dedce34d446d8b021d2e8bc3ebd64203adc876fc2913ba2d85d300cbc09cc
Sep 09 03:20:50.575712 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.575611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:50.669747 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.669699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:50.927626 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:50.927591    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:51.027814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.027779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:51.098760 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.098722    2562 patch_prober.go:28] interesting pod/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 09 03:20:51.098930 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.098797    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 09 03:20:51.128239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.128203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:51.318253 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.318220    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:51.433425 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.433391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v" event={"ID":"67d5ed21-4e4a-4287-a01c-0a3bb93e3637","Type":"ContainerStarted","Data":"2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f"}
Sep 09 03:20:51.433425 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.433442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v" event={"ID":"67d5ed21-4e4a-4287-a01c-0a3bb93e3637","Type":"ContainerStarted","Data":"fe9dedce34d446d8b021d2e8bc3ebd64203adc876fc2913ba2d85d300cbc09cc"}
Sep 09 03:20:51.433757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.433481    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:51.453320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.453247    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v" podStartSLOduration=2.110847901 podStartE2EDuration="2.453222828s" totalImagesPullingTime="342.374927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:51.450319638 +0000 UTC m=+900.012646207" watchObservedRunningTime="2026-09-09 03:20:51.453222828 +0000 UTC m=+900.015549374"
Sep 09 03:20:51.974009 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:51.973975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:52.098269 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:52.098235    2562 patch_prober.go:28] interesting pod/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 09 03:20:52.098479 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:52.098319    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 09 03:20:52.871224 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:52.871180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:53.099015 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:53.098968    2562 patch_prober.go:28] interesting pod/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 09 03:20:53.099437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:53.099027    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 09 03:20:53.708883 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:53.708842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:54.140600 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:54.140564    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:54.141046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:54.141000    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:54.142401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:54.142367    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:20:54.456748 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:54.456650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:57.220877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:57.220837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:57.323074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:57.323031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:57.446532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:57.446502    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:20:58.622434 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:58.622387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:59.357912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:59.357871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:59.964845 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:59.964800    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"]
Sep 09 03:20:59.965405 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:59.965065    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8" podUID="df19839c-9823-45b4-b2b7-116f930526b3" containerName="maas-api" containerID="cri-o://c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39" gracePeriod=30
Sep 09 03:20:59.994421 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:20:59.994374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:00.155034 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.154935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:00.220058 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.219988    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:21:00.347271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.347232    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-metrics-certs\") pod \"df19839c-9823-45b4-b2b7-116f930526b3\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") "
Sep 09 03:21:00.347470 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.347369    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19839c-9823-45b4-b2b7-116f930526b3-openshift-service-ca\") pod \"df19839c-9823-45b4-b2b7-116f930526b3\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") "
Sep 09 03:21:00.347574 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.347478    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls\") pod \"df19839c-9823-45b4-b2b7-116f930526b3\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") "
Sep 09 03:21:00.347574 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.347516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btcb4\" (UniqueName: \"kubernetes.io/projected/df19839c-9823-45b4-b2b7-116f930526b3-kube-api-access-btcb4\") pod \"df19839c-9823-45b4-b2b7-116f930526b3\" (UID: \"df19839c-9823-45b4-b2b7-116f930526b3\") "
Sep 09 03:21:00.347787 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.347758    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df19839c-9823-45b4-b2b7-116f930526b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df19839c-9823-45b4-b2b7-116f930526b3" (UID: "df19839c-9823-45b4-b2b7-116f930526b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:00.347955 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.347938    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19839c-9823-45b4-b2b7-116f930526b3-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:00.350137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.350105    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df19839c-9823-45b4-b2b7-116f930526b3" (UID: "df19839c-9823-45b4-b2b7-116f930526b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:21:00.350268 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.350212    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df19839c-9823-45b4-b2b7-116f930526b3-kube-api-access-btcb4" (OuterVolumeSpecName: "kube-api-access-btcb4") pod "df19839c-9823-45b4-b2b7-116f930526b3" (UID: "df19839c-9823-45b4-b2b7-116f930526b3"). InnerVolumeSpecName "kube-api-access-btcb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:00.350268 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.350246    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df19839c-9823-45b4-b2b7-116f930526b3" (UID: "df19839c-9823-45b4-b2b7-116f930526b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:21:00.376057 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.376027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:00.449476 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.449434    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:00.449476 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.449478    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcb4\" (UniqueName: \"kubernetes.io/projected/df19839c-9823-45b4-b2b7-116f930526b3-kube-api-access-btcb4\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:00.449742 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.449495    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df19839c-9823-45b4-b2b7-116f930526b3-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:00.479755 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.479665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:00.497715 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.497685    2562 generic.go:356] "Generic (PLEG): container finished" podID="df19839c-9823-45b4-b2b7-116f930526b3" containerID="c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39" exitCode=0
Sep 09 03:21:00.497855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.497734    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"
Sep 09 03:21:00.497855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.497751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8" event={"ID":"df19839c-9823-45b4-b2b7-116f930526b3","Type":"ContainerDied","Data":"c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39"}
Sep 09 03:21:00.497855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.497809    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8" event={"ID":"df19839c-9823-45b4-b2b7-116f930526b3","Type":"ContainerDied","Data":"043580ff81e6fcdb513c191f3262fb12712e486ef08a9204a1be8d65a880eeb0"}
Sep 09 03:21:00.497855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.497834    2562 scope.go:117] "RemoveContainer" containerID="c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39"
Sep 09 03:21:00.508393 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.508336    2562 scope.go:117] "RemoveContainer" containerID="c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39"
Sep 09 03:21:00.508688 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:21:00.508665    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39\": container with ID starting with c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39 not found: ID does not exist" containerID="c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39"
Sep 09 03:21:00.508781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.508696    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39"} err="failed to get container status \"c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39\": rpc error: code = NotFound desc = could not find container \"c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39\": container with ID starting with c4a88985cc9f0635dfff9e0c8a581eb4f2216e9dbe35d0ab4508ae42bdeb1d39 not found: ID does not exist"
Sep 09 03:21:00.519443 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.519417    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"]
Sep 09 03:21:00.525048 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.525020    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ecf63f7-59c8cc7bdd-9hcw8"]
Sep 09 03:21:00.556995 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.556952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:00.648172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.648133    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"]
Sep 09 03:21:00.648457 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.648425    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy" containerID="cri-o://21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994" gracePeriod=30
Sep 09 03:21:00.666066 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:00.666037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:01.176728 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:01.176690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:01.323529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:01.323494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:01.422762 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:01.422721    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:01.669097 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:01.669061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:01.947752 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:01.947656    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df19839c-9823-45b4-b2b7-116f930526b3" path="/var/lib/kubelet/pods/df19839c-9823-45b4-b2b7-116f930526b3/volumes"
Sep 09 03:21:01.994692 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:01.994650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:04.453478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.453443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:04.553452 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.553415    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"]
Sep 09 03:21:04.554079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.554061    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df19839c-9823-45b4-b2b7-116f930526b3" containerName="maas-api"
Sep 09 03:21:04.554134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.554083    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="df19839c-9823-45b4-b2b7-116f930526b3" containerName="maas-api"
Sep 09 03:21:04.554212 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.554202    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="df19839c-9823-45b4-b2b7-116f930526b3" containerName="maas-api"
Sep 09 03:21:04.557472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.557447    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.559721 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.559639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:04.559721 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.559662    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0e6f9624-serving-cert\""
Sep 09 03:21:04.576489 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.576449    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"]
Sep 09 03:21:04.587356 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.587322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47913973-0df4-451a-bdf8-2e41a5f67a77-openshift-service-ca\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.587555 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.587389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.587555 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.587487    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-metrics-certs\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.587686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.587593    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spkh\" (UniqueName: \"kubernetes.io/projected/47913973-0df4-451a-bdf8-2e41a5f67a77-kube-api-access-7spkh\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.652446 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.651776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:04.689047 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.689008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spkh\" (UniqueName: \"kubernetes.io/projected/47913973-0df4-451a-bdf8-2e41a5f67a77-kube-api-access-7spkh\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.689252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.689073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47913973-0df4-451a-bdf8-2e41a5f67a77-openshift-service-ca\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.689252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.689119    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.689252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.689156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-metrics-certs\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.689383 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:21:04.689248    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-serving-cert: secret "maas-api-e2e-migrate-0e6f9624-serving-cert" not found
Sep 09 03:21:04.689383 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:21:04.689330    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls podName:47913973-0df4-451a-bdf8-2e41a5f67a77 nodeName:}" failed. No retries permitted until 2026-09-09 03:21:05.189308211 +0000 UTC m=+913.751634752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls") pod "maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx" (UID: "47913973-0df4-451a-bdf8-2e41a5f67a77") : secret "maas-api-e2e-migrate-0e6f9624-serving-cert" not found
Sep 09 03:21:04.689961 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.689939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47913973-0df4-451a-bdf8-2e41a5f67a77-openshift-service-ca\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.691908 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.691883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-metrics-certs\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.698595 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.698568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spkh\" (UniqueName: \"kubernetes.io/projected/47913973-0df4-451a-bdf8-2e41a5f67a77-kube-api-access-7spkh\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:04.755067 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:04.754937    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:05.132764 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:05.132712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:05.195407 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:05.195366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:05.198141 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:05.198112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls\") pod \"maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:05.227351 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:05.227316    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:05.472272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:05.472178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:05.755817 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:05.755783    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"]
Sep 09 03:21:05.757770 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:21:05.757742    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47913973_0df4_451a_bdf8_2e41a5f67a77.slice/crio-4b70e55db5fa7906361e8f9124574ac6d726448a74d2aa54e1fc286380711a3d WatchSource:0}: Error finding container 4b70e55db5fa7906361e8f9124574ac6d726448a74d2aa54e1fc286380711a3d: Status 404 returned error can't find the container with id 4b70e55db5fa7906361e8f9124574ac6d726448a74d2aa54e1fc286380711a3d
Sep 09 03:21:05.926463 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:05.926437    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:21:06.005158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005064    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-envoy\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005100    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-certs\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005153    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dthkq\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-kube-api-access-dthkq\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-credential-socket\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005252    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-socket\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005306    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-istio-token\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005356    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc284174-f17e-4f26-9ba7-723976502a7e-istiod-ca-cert\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005387    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-data\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005730 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc284174-f17e-4f26-9ba7-723976502a7e-istio-podinfo\") pod \"cc284174-f17e-4f26-9ba7-723976502a7e\" (UID: \"cc284174-f17e-4f26-9ba7-723976502a7e\") "
Sep 09 03:21:06.005730 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005661    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:06.005955 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.005931    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:06.006197 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.006173    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.007106 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.006869    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:06.007106 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.007018    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc284174-f17e-4f26-9ba7-723976502a7e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:06.007983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.007960    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:06.009478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.009439    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-kube-api-access-dthkq" (OuterVolumeSpecName: "kube-api-access-dthkq") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "kube-api-access-dthkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:06.009635 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.009496    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:06.021951 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.021905    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc284174-f17e-4f26-9ba7-723976502a7e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:21:06.021951 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.021926    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc284174-f17e-4f26-9ba7-723976502a7e" (UID: "cc284174-f17e-4f26-9ba7-723976502a7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:06.106759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.106723    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.106759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.106754    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc284174-f17e-4f26-9ba7-723976502a7e-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.106759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.106762    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.106759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.106770    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc284174-f17e-4f26-9ba7-723976502a7e-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.107044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.106779    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.107044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.106787    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dthkq\" (UniqueName: \"kubernetes.io/projected/cc284174-f17e-4f26-9ba7-723976502a7e-kube-api-access-dthkq\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.107044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.106794    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.107044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.106803    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc284174-f17e-4f26-9ba7-723976502a7e-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:06.519451 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.519410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:06.542987 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.542932    2562 generic.go:356] "Generic (PLEG): container finished" podID="cc284174-f17e-4f26-9ba7-723976502a7e" containerID="21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994" exitCode=0
Sep 09 03:21:06.543294 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.543265    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"
Sep 09 03:21:06.547148 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.547113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" event={"ID":"cc284174-f17e-4f26-9ba7-723976502a7e","Type":"ContainerDied","Data":"21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994"}
Sep 09 03:21:06.547313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.547224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7" event={"ID":"cc284174-f17e-4f26-9ba7-723976502a7e","Type":"ContainerDied","Data":"a3be5d9e485a8f84e76f3742214a94be98944ae7e88d6cd9d85898ed0667b148"}
Sep 09 03:21:06.547313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.547261    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:06.547313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.547288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx" event={"ID":"47913973-0df4-451a-bdf8-2e41a5f67a77","Type":"ContainerStarted","Data":"1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965"}
Sep 09 03:21:06.547313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.547311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx" event={"ID":"47913973-0df4-451a-bdf8-2e41a5f67a77","Type":"ContainerStarted","Data":"4b70e55db5fa7906361e8f9124574ac6d726448a74d2aa54e1fc286380711a3d"}
Sep 09 03:21:06.547588 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.547346    2562 scope.go:117] "RemoveContainer" containerID="21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994"
Sep 09 03:21:06.561799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.561730    2562 scope.go:117] "RemoveContainer" containerID="96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556"
Sep 09 03:21:06.566965 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.566902    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx" podStartSLOduration=2.18214596 podStartE2EDuration="2.566880545s" totalImagesPullingTime="384.734585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:06.56493157 +0000 UTC m=+915.127258128" watchObservedRunningTime="2026-09-09 03:21:06.566880545 +0000 UTC m=+915.129207091"
Sep 09 03:21:06.575981 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.575958    2562 scope.go:117] "RemoveContainer" containerID="21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994"
Sep 09 03:21:06.576347 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:21:06.576325    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994\": container with ID starting with 21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994 not found: ID does not exist" containerID="21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994"
Sep 09 03:21:06.576441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.576373    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994"} err="failed to get container status \"21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994\": rpc error: code = NotFound desc = could not find container \"21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994\": container with ID starting with 21778140573fe7922f7c249c15d4b34049c3e9665faf97d20fec40bfa249e994 not found: ID does not exist"
Sep 09 03:21:06.576441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.576390    2562 scope.go:117] "RemoveContainer" containerID="96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556"
Sep 09 03:21:06.576735 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:21:06.576702    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556\": container with ID starting with 96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556 not found: ID does not exist" containerID="96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556"
Sep 09 03:21:06.576819 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.576736    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556"} err="failed to get container status \"96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556\": rpc error: code = NotFound desc = could not find container \"96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556\": container with ID starting with 96e2111a10aef2078b1be7ac01629a72b86edd764c19bd399c3705f35d3dd556 not found: ID does not exist"
Sep 09 03:21:06.584734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.584702    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"]
Sep 09 03:21:06.588670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:06.588646    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5ecf63f7-openshift-default-5f7ddd4fdb-tw2z7"]
Sep 09 03:21:07.938512 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:07.938475    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" path="/var/lib/kubelet/pods/cc284174-f17e-4f26-9ba7-723976502a7e/volumes"
Sep 09 03:21:11.001082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:11.001047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:12.559535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:12.559504    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:13.716324 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:13.716288    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:25.115000 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:25.114957    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:25.216868 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:25.216830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:25.648190 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:25.648154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:30.670624 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:30.670531    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:30.722263 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:30.722220    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:35.214598 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.214559    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"]
Sep 09 03:21:35.215132 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.214846    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx" podUID="47913973-0df4-451a-bdf8-2e41a5f67a77" containerName="maas-api" containerID="cri-o://1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965" gracePeriod=30
Sep 09 03:21:35.246387 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.246341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:35.309534 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.309501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:35.371111 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.371067    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:35.443528 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.443458    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:35.480474 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.480389    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:35.511593 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.511488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:35.592357 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.592326    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:35.614242 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.614202    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-metrics-certs\") pod \"47913973-0df4-451a-bdf8-2e41a5f67a77\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") "
Sep 09 03:21:35.614242 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.614246    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls\") pod \"47913973-0df4-451a-bdf8-2e41a5f67a77\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") "
Sep 09 03:21:35.614506 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.614284    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47913973-0df4-451a-bdf8-2e41a5f67a77-openshift-service-ca\") pod \"47913973-0df4-451a-bdf8-2e41a5f67a77\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") "
Sep 09 03:21:35.614506 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.614358    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7spkh\" (UniqueName: \"kubernetes.io/projected/47913973-0df4-451a-bdf8-2e41a5f67a77-kube-api-access-7spkh\") pod \"47913973-0df4-451a-bdf8-2e41a5f67a77\" (UID: \"47913973-0df4-451a-bdf8-2e41a5f67a77\") "
Sep 09 03:21:35.614754 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.614724    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47913973-0df4-451a-bdf8-2e41a5f67a77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47913973-0df4-451a-bdf8-2e41a5f67a77" (UID: "47913973-0df4-451a-bdf8-2e41a5f67a77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:35.616851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.616823    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47913973-0df4-451a-bdf8-2e41a5f67a77-kube-api-access-7spkh" (OuterVolumeSpecName: "kube-api-access-7spkh") pod "47913973-0df4-451a-bdf8-2e41a5f67a77" (UID: "47913973-0df4-451a-bdf8-2e41a5f67a77"). InnerVolumeSpecName "kube-api-access-7spkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:35.616946 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.616876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "47913973-0df4-451a-bdf8-2e41a5f67a77" (UID: "47913973-0df4-451a-bdf8-2e41a5f67a77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:21:35.616946 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.616926    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47913973-0df4-451a-bdf8-2e41a5f67a77" (UID: "47913973-0df4-451a-bdf8-2e41a5f67a77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:21:35.715427 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.715377    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:35.715427 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.715415    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47913973-0df4-451a-bdf8-2e41a5f67a77-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:35.715427 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.715424    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47913973-0df4-451a-bdf8-2e41a5f67a77-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:35.715427 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.715433    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7spkh\" (UniqueName: \"kubernetes.io/projected/47913973-0df4-451a-bdf8-2e41a5f67a77-kube-api-access-7spkh\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:21:35.737559 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.737437    2562 generic.go:356] "Generic (PLEG): container finished" podID="47913973-0df4-451a-bdf8-2e41a5f67a77" containerID="1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965" exitCode=0
Sep 09 03:21:35.737559 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.737480    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"
Sep 09 03:21:35.737559 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.737522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx" event={"ID":"47913973-0df4-451a-bdf8-2e41a5f67a77","Type":"ContainerDied","Data":"1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965"}
Sep 09 03:21:35.737840 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.737609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx" event={"ID":"47913973-0df4-451a-bdf8-2e41a5f67a77","Type":"ContainerDied","Data":"4b70e55db5fa7906361e8f9124574ac6d726448a74d2aa54e1fc286380711a3d"}
Sep 09 03:21:35.737840 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.737636    2562 scope.go:117] "RemoveContainer" containerID="1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965"
Sep 09 03:21:35.749495 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.749467    2562 scope.go:117] "RemoveContainer" containerID="1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965"
Sep 09 03:21:35.749996 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:21:35.749952    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965\": container with ID starting with 1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965 not found: ID does not exist" containerID="1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965"
Sep 09 03:21:35.749996 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.749985    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965"} err="failed to get container status \"1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965\": rpc error: code = NotFound desc = could not find container \"1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965\": container with ID starting with 1c2e6ddcc7b200f79e803e8169f21e409b0e6bc8ed65297a6476dd6304ead965 not found: ID does not exist"
Sep 09 03:21:35.759460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.759429    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"]
Sep 09 03:21:35.763391 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.763364    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e6f9624-f6c49b496-4hgbx"]
Sep 09 03:21:35.938496 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:35.938455    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47913973-0df4-451a-bdf8-2e41a5f67a77" path="/var/lib/kubelet/pods/47913973-0df4-451a-bdf8-2e41a5f67a77/volumes"
Sep 09 03:21:36.013220 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.013104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:36.280515 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.280420    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"]
Sep 09 03:21:36.280993 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.280973    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy"
Sep 09 03:21:36.281059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.280994    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy"
Sep 09 03:21:36.281059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.281005    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47913973-0df4-451a-bdf8-2e41a5f67a77" containerName="maas-api"
Sep 09 03:21:36.281059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.281011    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="47913973-0df4-451a-bdf8-2e41a5f67a77" containerName="maas-api"
Sep 09 03:21:36.281208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.281095    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="47913973-0df4-451a-bdf8-2e41a5f67a77" containerName="maas-api"
Sep 09 03:21:36.281208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.281109    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy"
Sep 09 03:21:36.281208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.281115    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy"
Sep 09 03:21:36.286069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.286045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.288365 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.288300    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-189a872e-openshift-default-dockercfg-6b2hq\""
Sep 09 03:21:36.298266 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.298226    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"]
Sep 09 03:21:36.311619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.311586    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:36.322257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322229    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-token\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.322403 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-envoy\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.322403 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istiod-ca-cert\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.322495 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322399    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-credential-socket\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.322495 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-podinfo\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.322495 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322476    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdrtq\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-kube-api-access-jdrtq\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.322651 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-data\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.322651 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-certs\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.322740 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.322650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-socket\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423411 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdrtq\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-kube-api-access-jdrtq\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-data\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-certs\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-socket\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-token\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-envoy\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istiod-ca-cert\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-credential-socket\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-podinfo\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.423941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.423925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-data\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.424149 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.424067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-socket\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.424149 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.424111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-certs\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.429753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.424667    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-credential-socket\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.429753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.424915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istiod-ca-cert\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.429753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.427724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-podinfo\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.434463 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.434306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-token\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.434962 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.434762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdrtq\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-kube-api-access-jdrtq\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.435905 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.435880    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-envoy\") pod \"e2e-ait-189a872e-openshift-default-986cfb94-xmvn7\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") " pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.491756 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.491710    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:36.598792 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.598757    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:36.880427 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.880395    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"]
Sep 09 03:21:36.882196 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:21:36.882167    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b52855b_7dfe_4b8d_b41b_0672af5f657c.slice/crio-09e38e39a8fe1f7ae68da676c1b74376031bfa9bddbf83c8bbef029b0a77e9f7 WatchSource:0}: Error finding container 09e38e39a8fe1f7ae68da676c1b74376031bfa9bddbf83c8bbef029b0a77e9f7: Status 404 returned error can't find the container with id 09e38e39a8fe1f7ae68da676c1b74376031bfa9bddbf83c8bbef029b0a77e9f7
Sep 09 03:21:36.884458 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.884423    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:21:36.884528 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.884495    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:21:36.884588 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:36.884526    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:21:37.757462 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:37.757427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" event={"ID":"7b52855b-7dfe-4b8d-b41b-0672af5f657c","Type":"ContainerStarted","Data":"eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274"}
Sep 09 03:21:37.757971 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:37.757483    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" event={"ID":"7b52855b-7dfe-4b8d-b41b-0672af5f657c","Type":"ContainerStarted","Data":"09e38e39a8fe1f7ae68da676c1b74376031bfa9bddbf83c8bbef029b0a77e9f7"}
Sep 09 03:21:37.778166 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:37.778102    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podStartSLOduration=1.778082573 podStartE2EDuration="1.778082573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:37.775258706 +0000 UTC m=+946.337585263" watchObservedRunningTime="2026-09-09 03:21:37.778082573 +0000 UTC m=+946.340409130"
Sep 09 03:21:38.599404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:38.599358    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:38.605167 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:38.605138    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:38.764599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:38.764564    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:38.765663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:38.765646    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:39.134691 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:39.134654    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:39.236695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:39.236657    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:40.337151 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.337115    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"]
Sep 09 03:21:40.337680 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.337636    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy"
Sep 09 03:21:40.337680 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.337653    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc284174-f17e-4f26-9ba7-723976502a7e" containerName="istio-proxy"
Sep 09 03:21:40.341165 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.341141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.346169 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.346137    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-189a872e-serving-cert\""
Sep 09 03:21:40.355040 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.354993    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"]
Sep 09 03:21:40.360749 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.360717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.361031 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.361001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc224377-46b0-4f0b-b737-32c16cba067a-openshift-service-ca\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.361147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.361101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-metrics-certs\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.361218 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.361150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn9hp\" (UniqueName: \"kubernetes.io/projected/cc224377-46b0-4f0b-b737-32c16cba067a-kube-api-access-hn9hp\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.383808 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.383772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:40.461877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.461830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc224377-46b0-4f0b-b737-32c16cba067a-openshift-service-ca\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.462071 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.461914    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-metrics-certs\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.462071 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.461943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9hp\" (UniqueName: \"kubernetes.io/projected/cc224377-46b0-4f0b-b737-32c16cba067a-kube-api-access-hn9hp\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.462071 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.461994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.462263 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:21:40.462089    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-serving-cert: secret "maas-api-e2e-ait-189a872e-serving-cert" not found
Sep 09 03:21:40.462263 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:21:40.462155    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls podName:cc224377-46b0-4f0b-b737-32c16cba067a nodeName:}" failed. No retries permitted until 2026-09-09 03:21:40.962138599 +0000 UTC m=+949.524465122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls") pod "maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw" (UID: "cc224377-46b0-4f0b-b737-32c16cba067a") : secret "maas-api-e2e-ait-189a872e-serving-cert" not found
Sep 09 03:21:40.462649 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.462624    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc224377-46b0-4f0b-b737-32c16cba067a-openshift-service-ca\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.464612 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.464591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-metrics-certs\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.483043 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.483015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9hp\" (UniqueName: \"kubernetes.io/projected/cc224377-46b0-4f0b-b737-32c16cba067a-kube-api-access-hn9hp\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.489847 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.489807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:40.588078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.587979    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:40.968322 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.968222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:40.970920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:40.970894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls\") pod \"maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:41.025216 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:41.025174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:41.111271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:41.111221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:41.261493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:41.261389    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:41.545268 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:41.545238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"]
Sep 09 03:21:41.783329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:41.783284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw" event={"ID":"cc224377-46b0-4f0b-b737-32c16cba067a","Type":"ContainerStarted","Data":"a746a48831dd4bcd779959c6ab26e99cf534e4ab7ecf987f55b64cc35afe4059"}
Sep 09 03:21:42.791218 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:42.791182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw" event={"ID":"cc224377-46b0-4f0b-b737-32c16cba067a","Type":"ContainerStarted","Data":"06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c"}
Sep 09 03:21:42.791721 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:42.791294    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:42.810116 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:42.810058    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw" podStartSLOduration=2.358615863 podStartE2EDuration="2.810040434s" totalImagesPullingTime="451.424571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:42.807299286 +0000 UTC m=+951.369625853" watchObservedRunningTime="2026-09-09 03:21:42.810040434 +0000 UTC m=+951.372366978"
Sep 09 03:21:48.803568 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:48.803513    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:21:52.619292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:52.619259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:54.805749 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:54.805712    2562 patch_prober.go:28] interesting pod/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 03:21:54.806167 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:54.805771    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 03:21:56.888458 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:56.888423    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerID="eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274" exitCode=0
Sep 09 03:21:56.889060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:56.888497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" event={"ID":"7b52855b-7dfe-4b8d-b41b-0672af5f657c","Type":"ContainerDied","Data":"eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274"}
Sep 09 03:21:56.889132 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:56.889081    2562 scope.go:117] "RemoveContainer" containerID="eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274"
Sep 09 03:21:56.890530 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:56.890493    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:21:56.890672 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:56.890603    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:21:56.890672 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:56.890640    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:21:57.897378 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:57.897345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" event={"ID":"7b52855b-7dfe-4b8d-b41b-0672af5f657c","Type":"ContainerStarted","Data":"222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198"}
Sep 09 03:21:58.599282 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:58.599239    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:21:58.599533 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:58.599510    2562 patch_prober.go:28] interesting pod/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 03:21:58.599632 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:58.599598    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 03:21:59.599474 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:59.599429    2562 patch_prober.go:28] interesting pod/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 03:21:59.599923 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:21:59.599500    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 03:22:00.599793 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:00.599757    2562 patch_prober.go:28] interesting pod/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 03:22:00.600269 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:00.600245    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 03:22:01.599531 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:01.599488    2562 patch_prober.go:28] interesting pod/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 03:22:01.599762 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:01.599574    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 03:22:02.599656 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:02.599615    2562 patch_prober.go:28] interesting pod/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 03:22:02.600056 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:02.599676    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 03:22:03.600157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:03.600120    2562 patch_prober.go:28] interesting pod/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 03:22:03.600638 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:03.600182    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 03:22:04.603738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:04.603702    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:22:04.604215 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:04.604126    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:22:04.604765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:04.604743    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:22:10.983166 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:10.983129    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"]
Sep 09 03:22:10.983667 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:10.983359    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw" podUID="cc224377-46b0-4f0b-b737-32c16cba067a" containerName="maas-api" containerID="cri-o://06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c" gracePeriod=30
Sep 09 03:22:11.013450 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.013417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:11.110133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.110092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:11.187426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.187388    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:11.249496 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.249428    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:22:11.282609 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.282497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:11.308722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.308693    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc224377-46b0-4f0b-b737-32c16cba067a-openshift-service-ca\") pod \"cc224377-46b0-4f0b-b737-32c16cba067a\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") "
Sep 09 03:22:11.308912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.308780    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-metrics-certs\") pod \"cc224377-46b0-4f0b-b737-32c16cba067a\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") "
Sep 09 03:22:11.308912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.308853    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn9hp\" (UniqueName: \"kubernetes.io/projected/cc224377-46b0-4f0b-b737-32c16cba067a-kube-api-access-hn9hp\") pod \"cc224377-46b0-4f0b-b737-32c16cba067a\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") "
Sep 09 03:22:11.308912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.308887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls\") pod \"cc224377-46b0-4f0b-b737-32c16cba067a\" (UID: \"cc224377-46b0-4f0b-b737-32c16cba067a\") "
Sep 09 03:22:11.309145 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.309121    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc224377-46b0-4f0b-b737-32c16cba067a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc224377-46b0-4f0b-b737-32c16cba067a" (UID: "cc224377-46b0-4f0b-b737-32c16cba067a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:11.309410 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.309379    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc224377-46b0-4f0b-b737-32c16cba067a-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:11.311292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.311259    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc224377-46b0-4f0b-b737-32c16cba067a" (UID: "cc224377-46b0-4f0b-b737-32c16cba067a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:22:11.311728 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.311706    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc224377-46b0-4f0b-b737-32c16cba067a-kube-api-access-hn9hp" (OuterVolumeSpecName: "kube-api-access-hn9hp") pod "cc224377-46b0-4f0b-b737-32c16cba067a" (UID: "cc224377-46b0-4f0b-b737-32c16cba067a"). InnerVolumeSpecName "kube-api-access-hn9hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:11.311814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.311739    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc224377-46b0-4f0b-b737-32c16cba067a" (UID: "cc224377-46b0-4f0b-b737-32c16cba067a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:22:11.347144 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.347105    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:11.410134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.410081    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn9hp\" (UniqueName: \"kubernetes.io/projected/cc224377-46b0-4f0b-b737-32c16cba067a-kube-api-access-hn9hp\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:11.410134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.410135    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:11.410134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.410154    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc224377-46b0-4f0b-b737-32c16cba067a-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:11.987923 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.987885    2562 generic.go:356] "Generic (PLEG): container finished" podID="cc224377-46b0-4f0b-b737-32c16cba067a" containerID="06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c" exitCode=0
Sep 09 03:22:11.988450 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.987933    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"
Sep 09 03:22:11.988450 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.987975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw" event={"ID":"cc224377-46b0-4f0b-b737-32c16cba067a","Type":"ContainerDied","Data":"06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c"}
Sep 09 03:22:11.988450 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.988041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw" event={"ID":"cc224377-46b0-4f0b-b737-32c16cba067a","Type":"ContainerDied","Data":"a746a48831dd4bcd779959c6ab26e99cf534e4ab7ecf987f55b64cc35afe4059"}
Sep 09 03:22:11.988450 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:11.988077    2562 scope.go:117] "RemoveContainer" containerID="06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c"
Sep 09 03:22:12.001256 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:12.001232    2562 scope.go:117] "RemoveContainer" containerID="06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c"
Sep 09 03:22:12.001679 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:12.001649    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c\": container with ID starting with 06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c not found: ID does not exist" containerID="06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c"
Sep 09 03:22:12.001765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:12.001688    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c"} err="failed to get container status \"06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c\": rpc error: code = NotFound desc = could not find container \"06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c\": container with ID starting with 06fcf6a3a2be5e85748be311cb567f02b5cfaf7d66c63395b6204da9b6c6cf6c not found: ID does not exist"
Sep 09 03:22:12.009637 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:12.009599    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"]
Sep 09 03:22:12.014001 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:12.013976    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-189a872e-6f8cb9c984-t58lw"]
Sep 09 03:22:12.362379 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:12.362344    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"]
Sep 09 03:22:12.362643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:12.362613    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy" containerID="cri-o://222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198" gracePeriod=30
Sep 09 03:22:12.382021 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:12.381992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:12.829335 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:12.829301    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:13.203296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.203210    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"]
Sep 09 03:22:13.203926 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.203908    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc224377-46b0-4f0b-b737-32c16cba067a" containerName="maas-api"
Sep 09 03:22:13.203977 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.203929    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc224377-46b0-4f0b-b737-32c16cba067a" containerName="maas-api"
Sep 09 03:22:13.204038 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.204027    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc224377-46b0-4f0b-b737-32c16cba067a" containerName="maas-api"
Sep 09 03:22:13.207263 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.207242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.209491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.209464    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8f2b65e3-openshift-default-dockercfg-7c9rp\""
Sep 09 03:22:13.222564 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.222511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"]
Sep 09 03:22:13.227936 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.227889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:13.228127 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.227977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxsn2\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-kube-api-access-bxsn2\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.228127 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.228020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-socket\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.228127 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.228048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-token\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.228127 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.228067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-podinfo\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.228380 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.228154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istiod-ca-cert\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.228380 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.228223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-certs\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.228380 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.228259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-data\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.228380 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.228287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-envoy\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.228380 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.228342    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-credential-socket\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329651 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxsn2\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-kube-api-access-bxsn2\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329651 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-socket\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329651 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-token\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-podinfo\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istiod-ca-cert\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-certs\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-data\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-envoy\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.329992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.329951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-credential-socket\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.330316 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.330090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-socket\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.330316 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.330111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-certs\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.330316 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.330172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-data\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.330316 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.330273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-credential-socket\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.330528 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.330510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istiod-ca-cert\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.332654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.332415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-envoy\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.332814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.332759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-podinfo\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.346117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.346093    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-token\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.348532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.348503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxsn2\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-kube-api-access-bxsn2\") pod \"e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") " pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.391653 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.391613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:13.518757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.518669    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:13.804109 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.804079    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"]
Sep 09 03:22:13.806242 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:22:13.806218    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f119c2_6eac_4f11_b941_ebe4b44dcd37.slice/crio-366d211be952a2c5f3f78ac8d96784e92f252a1ad19eb6f5debe50ba0e9afb84 WatchSource:0}: Error finding container 366d211be952a2c5f3f78ac8d96784e92f252a1ad19eb6f5debe50ba0e9afb84: Status 404 returned error can't find the container with id 366d211be952a2c5f3f78ac8d96784e92f252a1ad19eb6f5debe50ba0e9afb84
Sep 09 03:22:13.808891 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.808855    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:22:13.808970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.808928    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:22:13.808970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.808955    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:22:13.940165 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:13.940126    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc224377-46b0-4f0b-b737-32c16cba067a" path="/var/lib/kubelet/pods/cc224377-46b0-4f0b-b737-32c16cba067a/volumes"
Sep 09 03:22:14.013471 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:14.013418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" event={"ID":"19f119c2-6eac-4f11-b941-ebe4b44dcd37","Type":"ContainerStarted","Data":"d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93"}
Sep 09 03:22:14.013679 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:14.013497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" event={"ID":"19f119c2-6eac-4f11-b941-ebe4b44dcd37","Type":"ContainerStarted","Data":"366d211be952a2c5f3f78ac8d96784e92f252a1ad19eb6f5debe50ba0e9afb84"}
Sep 09 03:22:14.047977 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:14.047926    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" podStartSLOduration=1.047908273 podStartE2EDuration="1.047908273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:22:14.044419464 +0000 UTC m=+982.606746019" watchObservedRunningTime="2026-09-09 03:22:14.047908273 +0000 UTC m=+982.610234818"
Sep 09 03:22:14.519500 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:14.519461    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:14.525097 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:14.525062    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:15.020611 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:15.020569    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:15.021712 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:15.021690    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:15.378488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:15.378453    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:15.473675 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:15.473638    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:16.522877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.522836    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"]
Sep 09 03:22:16.526418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.526393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.528961 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.528932    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8f2b65e3-serving-cert\""
Sep 09 03:22:16.539805 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.539661    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"]
Sep 09 03:22:16.563689 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.563654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62db1ede-5c57-4a6d-84f7-c7e0418769d3-openshift-service-ca\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.563869 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.563723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf82v\" (UniqueName: \"kubernetes.io/projected/62db1ede-5c57-4a6d-84f7-c7e0418769d3-kube-api-access-kf82v\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.563869 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.563816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-maas-api-tls\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.563991 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.563971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-metrics-certs\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.574248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.574214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:16.665492 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.665447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62db1ede-5c57-4a6d-84f7-c7e0418769d3-openshift-service-ca\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.665766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.665519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf82v\" (UniqueName: \"kubernetes.io/projected/62db1ede-5c57-4a6d-84f7-c7e0418769d3-kube-api-access-kf82v\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.665766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.665604    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-maas-api-tls\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.665766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.665722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-metrics-certs\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.666661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.666637    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62db1ede-5c57-4a6d-84f7-c7e0418769d3-openshift-service-ca\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.668461 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.668434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-maas-api-tls\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.668593 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.668463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-metrics-certs\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.672236 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.672185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:16.675072 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.675047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf82v\" (UniqueName: \"kubernetes.io/projected/62db1ede-5c57-4a6d-84f7-c7e0418769d3-kube-api-access-kf82v\") pod \"maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:16.770379 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.770348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:16.841565 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:16.841512    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:17.041217 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.041181    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:17.118911 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.118878    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"]
Sep 09 03:22:17.120087 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.120039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:17.120682 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:22:17.120658    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62db1ede_5c57_4a6d_84f7_c7e0418769d3.slice/crio-10e575637ea98a2e8123b9785122824e3cfd2e5a9cf51f733b7520fdbb89e3f2 WatchSource:0}: Error finding container 10e575637ea98a2e8123b9785122824e3cfd2e5a9cf51f733b7520fdbb89e3f2: Status 404 returned error can't find the container with id 10e575637ea98a2e8123b9785122824e3cfd2e5a9cf51f733b7520fdbb89e3f2
Sep 09 03:22:17.625882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.625812    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:22:17.675731 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.675691    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-credential-socket\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.675731 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.675732    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-token\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.675949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.675822    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdrtq\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-kube-api-access-jdrtq\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.675949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.675913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-data\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.675949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.675941    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istiod-ca-cert\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.676045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.675966    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-podinfo\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.676045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.676010    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-envoy\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.676045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.676036    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-certs\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.676137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.676063    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:17.676137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.676090    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-socket\") pod \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\" (UID: \"7b52855b-7dfe-4b8d-b41b-0672af5f657c\") "
Sep 09 03:22:17.677046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.676404    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:17.677046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.676826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:17.677046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.676935    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:17.677046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.677001    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:17.677046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.677025    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:17.677046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.677036    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:17.678877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.678854    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:17.679055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.679029    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:17.679458 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.679436    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:22:17.679597 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.679488    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-kube-api-access-jdrtq" (OuterVolumeSpecName: "kube-api-access-jdrtq") pod "7b52855b-7dfe-4b8d-b41b-0672af5f657c" (UID: "7b52855b-7dfe-4b8d-b41b-0672af5f657c"). InnerVolumeSpecName "kube-api-access-jdrtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:17.778182 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.778141    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdrtq\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-kube-api-access-jdrtq\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:17.778182 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.778176    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:17.778182 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.778185    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:17.778182 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.778194    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:17.778486 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.778202    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:17.778486 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.778210    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b52855b-7dfe-4b8d-b41b-0672af5f657c-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:17.778486 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:17.778217    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b52855b-7dfe-4b8d-b41b-0672af5f657c-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:18.040093 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.040053    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerID="222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198" exitCode=0
Sep 09 03:22:18.040301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.040112    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"
Sep 09 03:22:18.040301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.040135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" event={"ID":"7b52855b-7dfe-4b8d-b41b-0672af5f657c","Type":"ContainerDied","Data":"222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198"}
Sep 09 03:22:18.040301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.040190    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7" event={"ID":"7b52855b-7dfe-4b8d-b41b-0672af5f657c","Type":"ContainerDied","Data":"09e38e39a8fe1f7ae68da676c1b74376031bfa9bddbf83c8bbef029b0a77e9f7"}
Sep 09 03:22:18.040301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.040214    2562 scope.go:117] "RemoveContainer" containerID="222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198"
Sep 09 03:22:18.043321 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.043295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5" event={"ID":"62db1ede-5c57-4a6d-84f7-c7e0418769d3","Type":"ContainerStarted","Data":"3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b"}
Sep 09 03:22:18.043450 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.043353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5" event={"ID":"62db1ede-5c57-4a6d-84f7-c7e0418769d3","Type":"ContainerStarted","Data":"10e575637ea98a2e8123b9785122824e3cfd2e5a9cf51f733b7520fdbb89e3f2"}
Sep 09 03:22:18.055054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.055022    2562 scope.go:117] "RemoveContainer" containerID="eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274"
Sep 09 03:22:18.064633 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.064612    2562 scope.go:117] "RemoveContainer" containerID="222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198"
Sep 09 03:22:18.065300 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:18.065275    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198\": container with ID starting with 222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198 not found: ID does not exist" containerID="222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198"
Sep 09 03:22:18.065406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.065309    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198"} err="failed to get container status \"222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198\": rpc error: code = NotFound desc = could not find container \"222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198\": container with ID starting with 222193d336b793e25deb94be8ecc89f632a6f4c66c2858bedbbfeed113cb9198 not found: ID does not exist"
Sep 09 03:22:18.065406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.065328    2562 scope.go:117] "RemoveContainer" containerID="eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274"
Sep 09 03:22:18.065673 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:18.065645    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274\": container with ID starting with eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274 not found: ID does not exist" containerID="eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274"
Sep 09 03:22:18.065758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.065683    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274"} err="failed to get container status \"eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274\": rpc error: code = NotFound desc = could not find container \"eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274\": container with ID starting with eb3e8e4d61e5fbe733d319bd1b8db3f6b1490f98cd33dc58059bc30fa8b15274 not found: ID does not exist"
Sep 09 03:22:18.067917 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.067849    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5" podStartSLOduration=1.705072145 podStartE2EDuration="2.067833686s" totalImagesPullingTime="362.761541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:22:18.064376839 +0000 UTC m=+986.626703386" watchObservedRunningTime="2026-09-09 03:22:18.067833686 +0000 UTC m=+986.630160237"
Sep 09 03:22:18.084664 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.084632    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"]
Sep 09 03:22:18.089314 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:18.089284    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-189a872e-openshift-default-986cfb94-xmvn7"]
Sep 09 03:22:19.050822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:19.050798    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:19.939159 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:19.939123    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" path="/var/lib/kubelet/pods/7b52855b-7dfe-4b8d-b41b-0672af5f657c/volumes"
Sep 09 03:22:25.066869 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:25.066837    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:26.973254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:26.973197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:33.216079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.216038    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"]
Sep 09 03:22:33.220010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.217500    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy"
Sep 09 03:22:33.220010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.217567    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy"
Sep 09 03:22:33.220010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.217600    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy"
Sep 09 03:22:33.220010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.217609    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy"
Sep 09 03:22:33.220010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.217868    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy"
Sep 09 03:22:33.224252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.224136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.226596 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.226569    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-f99e31e6-kserve-self-signed-certs\""
Sep 09 03:22:33.230752 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.230722    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"]
Sep 09 03:22:33.338978 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.338938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-dshm\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.338978 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.338980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-model-cache\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.339246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.339001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd307ca1-bba4-499a-90e2-d82da526f489-tls-certs\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.339246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.339086    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-tmp-dir\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.339246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.339125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-home\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.339246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.339157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwc8\" (UniqueName: \"kubernetes.io/projected/fd307ca1-bba4-499a-90e2-d82da526f489-kube-api-access-bkwc8\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440024 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.439980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwc8\" (UniqueName: \"kubernetes.io/projected/fd307ca1-bba4-499a-90e2-d82da526f489-kube-api-access-bkwc8\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440215 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.440078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-dshm\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440215 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.440104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-model-cache\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440215 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.440124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd307ca1-bba4-499a-90e2-d82da526f489-tls-certs\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440215 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.440182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-tmp-dir\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440409 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.440241    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-home\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440675 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.440650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-model-cache\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.440686    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-home\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.440758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.440658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-tmp-dir\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.444147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.444119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd307ca1-bba4-499a-90e2-d82da526f489-tls-certs\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.450814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.450775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-dshm\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.452275 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.452240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwc8\" (UniqueName: \"kubernetes.io/projected/fd307ca1-bba4-499a-90e2-d82da526f489-kube-api-access-bkwc8\") pod \"e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") " pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.535784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.535749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:33.836951 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.836744    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:33.837308 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:33.837282    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"]
Sep 09 03:22:33.839135 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:22:33.839106    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd307ca1_bba4_499a_90e2_d82da526f489.slice/crio-87def3f2dfe52d53768e7c627b25ad0340da6a41fdc117b30b45eacb7a901bf2 WatchSource:0}: Error finding container 87def3f2dfe52d53768e7c627b25ad0340da6a41fdc117b30b45eacb7a901bf2: Status 404 returned error can't find the container with id 87def3f2dfe52d53768e7c627b25ad0340da6a41fdc117b30b45eacb7a901bf2
Sep 09 03:22:34.153937 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:34.153837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8" event={"ID":"fd307ca1-bba4-499a-90e2-d82da526f489","Type":"ContainerStarted","Data":"44c454f31333d4c266f1ccf88a26db2dc0511cb09c8ce7079c5edcef1533172a"}
Sep 09 03:22:34.153937 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:34.153897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8" event={"ID":"fd307ca1-bba4-499a-90e2-d82da526f489","Type":"ContainerStarted","Data":"87def3f2dfe52d53768e7c627b25ad0340da6a41fdc117b30b45eacb7a901bf2"}
Sep 09 03:22:34.173606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:34.173530    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8" podStartSLOduration=1.173511212 podStartE2EDuration="1.173511212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:22:34.170952178 +0000 UTC m=+1002.733278726" watchObservedRunningTime="2026-09-09 03:22:34.173511212 +0000 UTC m=+1002.735837757"
Sep 09 03:22:37.819132 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:37.819093    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:38.184293 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.184257    2562 generic.go:356] "Generic (PLEG): container finished" podID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerID="d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93" exitCode=0
Sep 09 03:22:38.184478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.184330    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" event={"ID":"19f119c2-6eac-4f11-b941-ebe4b44dcd37","Type":"ContainerDied","Data":"d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93"}
Sep 09 03:22:38.184802 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.184789    2562 scope.go:117] "RemoveContainer" containerID="d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93"
Sep 09 03:22:38.186046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.186010    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:22:38.186166 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.186088    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:22:38.186166 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.186120    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:22:38.831728 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.831690    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"]
Sep 09 03:22:38.832191 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.831910    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4" podUID="d3c77ffb-20ee-4c1f-a88b-856193ee8214" containerName="maas-api" containerID="cri-o://097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594" gracePeriod=30
Sep 09 03:22:38.864776 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:38.864740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:39.012499 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.012421    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:39.109673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.109639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:39.130259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.130223    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:22:39.197594 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.197527    2562 generic.go:356] "Generic (PLEG): container finished" podID="d3c77ffb-20ee-4c1f-a88b-856193ee8214" containerID="097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594" exitCode=0
Sep 09 03:22:39.197771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.197605    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"
Sep 09 03:22:39.197771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.197604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4" event={"ID":"d3c77ffb-20ee-4c1f-a88b-856193ee8214","Type":"ContainerDied","Data":"097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594"}
Sep 09 03:22:39.197771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.197703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4" event={"ID":"d3c77ffb-20ee-4c1f-a88b-856193ee8214","Type":"ContainerDied","Data":"07896f77ea571a30de7e3f223717f895b74f261a6c688d183afe5f45913ab9f9"}
Sep 09 03:22:39.197771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.197743    2562 scope.go:117] "RemoveContainer" containerID="097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594"
Sep 09 03:22:39.200795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.200747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" event={"ID":"19f119c2-6eac-4f11-b941-ebe4b44dcd37","Type":"ContainerStarted","Data":"e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f"}
Sep 09 03:22:39.207890 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.207868    2562 scope.go:117] "RemoveContainer" containerID="097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594"
Sep 09 03:22:39.208278 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:39.208255    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594\": container with ID starting with 097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594 not found: ID does not exist" containerID="097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594"
Sep 09 03:22:39.208369 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.208289    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594"} err="failed to get container status \"097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594\": rpc error: code = NotFound desc = could not find container \"097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594\": container with ID starting with 097d6c1fe7f83a9b8930ff4f4776f5ccce4f66c4d161e6376a64c3b418e66594 not found: ID does not exist"
Sep 09 03:22:39.214260 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.214226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:39.226531 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.226505    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls\") pod \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") "
Sep 09 03:22:39.226652 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.226582    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql6kk\" (UniqueName: \"kubernetes.io/projected/d3c77ffb-20ee-4c1f-a88b-856193ee8214-kube-api-access-ql6kk\") pod \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") "
Sep 09 03:22:39.226765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.226747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c77ffb-20ee-4c1f-a88b-856193ee8214-openshift-service-ca\") pod \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") "
Sep 09 03:22:39.226850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.226837    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-metrics-certs\") pod \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\" (UID: \"d3c77ffb-20ee-4c1f-a88b-856193ee8214\") "
Sep 09 03:22:39.227125 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.227100    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c77ffb-20ee-4c1f-a88b-856193ee8214-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3c77ffb-20ee-4c1f-a88b-856193ee8214" (UID: "d3c77ffb-20ee-4c1f-a88b-856193ee8214"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:39.227394 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.227267    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c77ffb-20ee-4c1f-a88b-856193ee8214-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:39.229678 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.229648    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3c77ffb-20ee-4c1f-a88b-856193ee8214" (UID: "d3c77ffb-20ee-4c1f-a88b-856193ee8214"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:22:39.229678 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.229654    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c77ffb-20ee-4c1f-a88b-856193ee8214-kube-api-access-ql6kk" (OuterVolumeSpecName: "kube-api-access-ql6kk") pod "d3c77ffb-20ee-4c1f-a88b-856193ee8214" (UID: "d3c77ffb-20ee-4c1f-a88b-856193ee8214"). InnerVolumeSpecName "kube-api-access-ql6kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:39.229862 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.229658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d3c77ffb-20ee-4c1f-a88b-856193ee8214" (UID: "d3c77ffb-20ee-4c1f-a88b-856193ee8214"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:22:39.328741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.328704    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:39.328741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.328736    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c77ffb-20ee-4c1f-a88b-856193ee8214-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:39.328741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.328745    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6kk\" (UniqueName: \"kubernetes.io/projected/d3c77ffb-20ee-4c1f-a88b-856193ee8214-kube-api-access-ql6kk\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:39.357637 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.356191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:39.519855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.519643    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:39.520065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.519887    2562 patch_prober.go:28] interesting pod/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 03:22:39.520065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.519943    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 03:22:39.523406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.523375    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"]
Sep 09 03:22:39.526114 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.526089    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-be414f-8586747548-582x4"]
Sep 09 03:22:39.724955 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.724849    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:39.940127 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:39.940092    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c77ffb-20ee-4c1f-a88b-856193ee8214" path="/var/lib/kubelet/pods/d3c77ffb-20ee-4c1f-a88b-856193ee8214/volumes"
Sep 09 03:22:40.519783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:40.519752    2562 patch_prober.go:28] interesting pod/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 03:22:40.519955 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:40.519820    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 03:22:41.519158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:41.519118    2562 patch_prober.go:28] interesting pod/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 03:22:41.519609 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:41.519178    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 03:22:42.519333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:42.519291    2562 patch_prober.go:28] interesting pod/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 03:22:42.519787 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:42.519365    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 03:22:43.520016 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:43.519980    2562 patch_prober.go:28] interesting pod/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 03:22:43.520420 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:43.520044    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 03:22:43.536046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:43.536017    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:43.536170 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:43.536065    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:43.576113 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:43.576080    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:44.272240 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:44.272204    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:22:44.519644 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:44.519608    2562 patch_prober.go:28] interesting pod/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 03:22:44.519827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:44.519670    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 03:22:44.796862 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:44.796816    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:45.021426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:45.021389    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:45.523697 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:45.523664    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:45.524999 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:45.524975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:47.160572 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.157084    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"]
Sep 09 03:22:47.160572 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.157422    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5" podUID="62db1ede-5c57-4a6d-84f7-c7e0418769d3" containerName="maas-api" containerID="cri-o://3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b" gracePeriod=30
Sep 09 03:22:47.189729 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.189696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:47.285602 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.285561    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:47.386060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.386024    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:47.425286 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.425219    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:47.464099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.464063    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:47.530521 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.530477    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62db1ede-5c57-4a6d-84f7-c7e0418769d3-openshift-service-ca\") pod \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") "
Sep 09 03:22:47.530725 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.530534    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf82v\" (UniqueName: \"kubernetes.io/projected/62db1ede-5c57-4a6d-84f7-c7e0418769d3-kube-api-access-kf82v\") pod \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") "
Sep 09 03:22:47.530725 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.530591    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-metrics-certs\") pod \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") "
Sep 09 03:22:47.530725 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.530650    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-maas-api-tls\") pod \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\" (UID: \"62db1ede-5c57-4a6d-84f7-c7e0418769d3\") "
Sep 09 03:22:47.530989 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.530956    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62db1ede-5c57-4a6d-84f7-c7e0418769d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62db1ede-5c57-4a6d-84f7-c7e0418769d3" (UID: "62db1ede-5c57-4a6d-84f7-c7e0418769d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:47.533032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.532995    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62db1ede-5c57-4a6d-84f7-c7e0418769d3" (UID: "62db1ede-5c57-4a6d-84f7-c7e0418769d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:22:47.533032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.533008    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62db1ede-5c57-4a6d-84f7-c7e0418769d3-kube-api-access-kf82v" (OuterVolumeSpecName: "kube-api-access-kf82v") pod "62db1ede-5c57-4a6d-84f7-c7e0418769d3" (UID: "62db1ede-5c57-4a6d-84f7-c7e0418769d3"). InnerVolumeSpecName "kube-api-access-kf82v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:47.533032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.533022    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62db1ede-5c57-4a6d-84f7-c7e0418769d3" (UID: "62db1ede-5c57-4a6d-84f7-c7e0418769d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:22:47.619800 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.619760    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"]
Sep 09 03:22:47.620077 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.620044    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy" containerID="cri-o://e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f" gracePeriod=30
Sep 09 03:22:47.632071 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.632034    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62db1ede-5c57-4a6d-84f7-c7e0418769d3-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:47.632071 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.632067    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf82v\" (UniqueName: \"kubernetes.io/projected/62db1ede-5c57-4a6d-84f7-c7e0418769d3-kube-api-access-kf82v\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:47.632071 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.632080    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:47.632320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.632090    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62db1ede-5c57-4a6d-84f7-c7e0418769d3-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:47.647293 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:47.647263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:48.234177 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.234083    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"]
Sep 09 03:22:48.234527 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.234504    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c77ffb-20ee-4c1f-a88b-856193ee8214" containerName="maas-api"
Sep 09 03:22:48.234527 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.234515    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c77ffb-20ee-4c1f-a88b-856193ee8214" containerName="maas-api"
Sep 09 03:22:48.234648 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.234559    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62db1ede-5c57-4a6d-84f7-c7e0418769d3" containerName="maas-api"
Sep 09 03:22:48.234648 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.234569    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="62db1ede-5c57-4a6d-84f7-c7e0418769d3" containerName="maas-api"
Sep 09 03:22:48.234648 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.234642    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b52855b-7dfe-4b8d-b41b-0672af5f657c" containerName="istio-proxy"
Sep 09 03:22:48.234745 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.234651    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="62db1ede-5c57-4a6d-84f7-c7e0418769d3" containerName="maas-api"
Sep 09 03:22:48.234745 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.234658    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c77ffb-20ee-4c1f-a88b-856193ee8214" containerName="maas-api"
Sep 09 03:22:48.237298 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.237278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.240979 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:48.240946    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-inf-a-b27dde\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-inf-a-b27dde\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 09 03:22:48.241091 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:48.240946    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-27h74\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-inf-a-b27dde\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-inf-a-b27dde\"/\"default-dockercfg-27h74\"" type="*v1.Secret"
Sep 09 03:22:48.241344 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:48.241317    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-model-b27dde-kserve-55fc87c79c-rbb69\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-inf-a-b27dde\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="8dbb1560-6861-4f17-aa36-394c8bdfeab3" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.241393 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:48.241374    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"test-model-b27dde-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-inf-a-b27dde\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-inf-a-b27dde\"/\"test-model-b27dde-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 09 03:22:48.242519 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.242498    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b27dde\"/\"openshift-service-ca.crt\""
Sep 09 03:22:48.257084 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.257051    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"]
Sep 09 03:22:48.262312 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.262282    2562 generic.go:356] "Generic (PLEG): container finished" podID="62db1ede-5c57-4a6d-84f7-c7e0418769d3" containerID="3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b" exitCode=0
Sep 09 03:22:48.262458 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.262334    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"
Sep 09 03:22:48.262458 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.262380    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5" event={"ID":"62db1ede-5c57-4a6d-84f7-c7e0418769d3","Type":"ContainerDied","Data":"3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b"}
Sep 09 03:22:48.262458 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.262438    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5" event={"ID":"62db1ede-5c57-4a6d-84f7-c7e0418769d3","Type":"ContainerDied","Data":"10e575637ea98a2e8123b9785122824e3cfd2e5a9cf51f733b7520fdbb89e3f2"}
Sep 09 03:22:48.262650 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.262467    2562 scope.go:117] "RemoveContainer" containerID="3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b"
Sep 09 03:22:48.273618 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.273593    2562 scope.go:117] "RemoveContainer" containerID="3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b"
Sep 09 03:22:48.273998 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:48.273975    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b\": container with ID starting with 3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b not found: ID does not exist" containerID="3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b"
Sep 09 03:22:48.274103 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.274006    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b"} err="failed to get container status \"3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b\": rpc error: code = NotFound desc = could not find container \"3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b\": container with ID starting with 3aca17429ec4281c207ea82d43ed038c36cc9ae6c356b74401a2e93fd0369d1b not found: ID does not exist"
Sep 09 03:22:48.338085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.338049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tmp-dir\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.338238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.338109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-dshm\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.338238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.338171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lwf\" (UniqueName: \"kubernetes.io/projected/8dbb1560-6861-4f17-aa36-394c8bdfeab3-kube-api-access-j9lwf\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.338238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.338221    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-home\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.338489 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.338265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-model-cache\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.338489 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.338314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tls-certs\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.364240 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.364194    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"]
Sep 09 03:22:48.387079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.387042    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8f2b65e3-785dd4fc97-n8sr5"]
Sep 09 03:22:48.438988 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.438942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tmp-dir\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.438988 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.439005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-dshm\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.439289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.439141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lwf\" (UniqueName: \"kubernetes.io/projected/8dbb1560-6861-4f17-aa36-394c8bdfeab3-kube-api-access-j9lwf\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.439289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.439205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-home\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.439289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.439262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-model-cache\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.439450 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.439335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tls-certs\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.439509 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.439473    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tmp-dir\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.439589 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.439513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-home\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.439826 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.439733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-model-cache\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.440653 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.440623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:48.441585 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.441534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-dshm\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:48.588766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:48.588733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:49.150640 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.150606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:49.155008 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.154983    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b27dde\"/\"test-model-b27dde-kserve-self-signed-certs\""
Sep 09 03:22:49.162768 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.162745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tls-certs\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:49.422289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.422198    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b27dde\"/\"kube-root-ca.crt\""
Sep 09 03:22:49.433885 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.433855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lwf\" (UniqueName: \"kubernetes.io/projected/8dbb1560-6861-4f17-aa36-394c8bdfeab3-kube-api-access-j9lwf\") pod \"test-model-b27dde-kserve-55fc87c79c-rbb69\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") " pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:49.597732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.597696    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b27dde\"/\"default-dockercfg-27h74\""
Sep 09 03:22:49.599563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.599515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:49.904824 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.904798    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"]
Sep 09 03:22:49.906855 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:22:49.906817    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dbb1560_6861_4f17_aa36_394c8bdfeab3.slice/crio-5213a5931fb9b4b6c1e4d5dd5f167a7815cba2832e4d73360deb7249a3678da3 WatchSource:0}: Error finding container 5213a5931fb9b4b6c1e4d5dd5f167a7815cba2832e4d73360deb7249a3678da3: Status 404 returned error can't find the container with id 5213a5931fb9b4b6c1e4d5dd5f167a7815cba2832e4d73360deb7249a3678da3
Sep 09 03:22:49.939790 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:49.939754    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62db1ede-5c57-4a6d-84f7-c7e0418769d3" path="/var/lib/kubelet/pods/62db1ede-5c57-4a6d-84f7-c7e0418769d3/volumes"
Sep 09 03:22:50.281180 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:50.281091    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69" event={"ID":"8dbb1560-6861-4f17-aa36-394c8bdfeab3","Type":"ContainerStarted","Data":"f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1"}
Sep 09 03:22:50.281180 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:50.281131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69" event={"ID":"8dbb1560-6861-4f17-aa36-394c8bdfeab3","Type":"ContainerStarted","Data":"5213a5931fb9b4b6c1e4d5dd5f167a7815cba2832e4d73360deb7249a3678da3"}
Sep 09 03:22:50.326753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:50.326697    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69" podStartSLOduration=2.326675426 podStartE2EDuration="2.326675426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:22:50.320753271 +0000 UTC m=+1018.883079815" watchObservedRunningTime="2026-09-09 03:22:50.326675426 +0000 UTC m=+1018.889001971"
Sep 09 03:22:52.983227 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:52.983198    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:53.088405 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088370    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-data\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088436    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-credential-socket\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088456    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-socket\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088523    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxsn2\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-kube-api-access-bxsn2\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088568    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-token\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088598    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-envoy\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088635    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istiod-ca-cert\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-podinfo\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088718    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-certs\") pod \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\" (UID: \"19f119c2-6eac-4f11-b941-ebe4b44dcd37\") "
Sep 09 03:22:53.088914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088745    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-data" (OuterVolumeSpecName: "istio-data") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:53.088914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088845    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:53.088914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.088902    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:53.089230 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.089091    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:53.089292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.089275    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.089344 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.089300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:53.089522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.089416    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.089522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.089446    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.091225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.091198    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:22:53.091225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.091207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-token" (OuterVolumeSpecName: "istio-token") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:53.091483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.091459    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:22:53.091574 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.091534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-kube-api-access-bxsn2" (OuterVolumeSpecName: "kube-api-access-bxsn2") pod "19f119c2-6eac-4f11-b941-ebe4b44dcd37" (UID: "19f119c2-6eac-4f11-b941-ebe4b44dcd37"). InnerVolumeSpecName "kube-api-access-bxsn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:53.190812 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.190769    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxsn2\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-kube-api-access-bxsn2\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.190812 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.190805    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.190812 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.190815    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.190812 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.190822    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.191094 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.190830    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f119c2-6eac-4f11-b941-ebe4b44dcd37-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.191094 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.190839    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f119c2-6eac-4f11-b941-ebe4b44dcd37-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:22:53.307229 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.307191    2562 generic.go:356] "Generic (PLEG): container finished" podID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerID="e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f" exitCode=0
Sep 09 03:22:53.307398 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.307261    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"
Sep 09 03:22:53.307398 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.307277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" event={"ID":"19f119c2-6eac-4f11-b941-ebe4b44dcd37","Type":"ContainerDied","Data":"e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f"}
Sep 09 03:22:53.307398 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.307372    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx" event={"ID":"19f119c2-6eac-4f11-b941-ebe4b44dcd37","Type":"ContainerDied","Data":"366d211be952a2c5f3f78ac8d96784e92f252a1ad19eb6f5debe50ba0e9afb84"}
Sep 09 03:22:53.307589 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.307403    2562 scope.go:117] "RemoveContainer" containerID="e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f"
Sep 09 03:22:53.318415 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.318396    2562 scope.go:117] "RemoveContainer" containerID="d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93"
Sep 09 03:22:53.328426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.328405    2562 scope.go:117] "RemoveContainer" containerID="e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f"
Sep 09 03:22:53.328772 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:53.328750    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f\": container with ID starting with e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f not found: ID does not exist" containerID="e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f"
Sep 09 03:22:53.328862 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.328782    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f"} err="failed to get container status \"e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f\": rpc error: code = NotFound desc = could not find container \"e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f\": container with ID starting with e2c857c1fe08a1d5d493301ea15a9ddb6b2fb48912a36e41450d730abf3a9b7f not found: ID does not exist"
Sep 09 03:22:53.328862 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.328805    2562 scope.go:117] "RemoveContainer" containerID="d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93"
Sep 09 03:22:53.329105 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:22:53.329089    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93\": container with ID starting with d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93 not found: ID does not exist" containerID="d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93"
Sep 09 03:22:53.329147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.329110    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93"} err="failed to get container status \"d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93\": rpc error: code = NotFound desc = could not find container \"d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93\": container with ID starting with d816919a7afb45c14fc9300d0a49015493f2c4ff56ec217ab5160194dee6bb93 not found: ID does not exist"
Sep 09 03:22:53.339198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.339134    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"]
Sep 09 03:22:53.349709 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.349672    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8f2b65e3-openshift-default-68f7797864-f9hwx"]
Sep 09 03:22:53.452659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.452608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:53.939882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:53.939847    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" path="/var/lib/kubelet/pods/19f119c2-6eac-4f11-b941-ebe4b44dcd37/volumes"
Sep 09 03:22:55.210888 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:55.210844    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"]
Sep 09 03:22:55.330593 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:55.330557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:59.600619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:59.600574    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:59.601049 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:59.600634    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:22:59.613418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:22:59.613394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:23:00.270846 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:00.270807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:00.365807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:00.365775    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:23:06.276255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:06.276223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:06.377662 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:06.377616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:06.475869 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:06.475831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:06.576086 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:06.576027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:06.937715 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:06.937628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:10.233838 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.233773    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8" podUID="fd307ca1-bba4-499a-90e2-d82da526f489" containerName="main" containerID="cri-o://44c454f31333d4c266f1ccf88a26db2dc0511cb09c8ce7079c5edcef1533172a" gracePeriod=45
Sep 09 03:23:10.425638 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.425447    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd307ca1-bba4-499a-90e2-d82da526f489" containerID="44c454f31333d4c266f1ccf88a26db2dc0511cb09c8ce7079c5edcef1533172a" exitCode=0
Sep 09 03:23:10.425638 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.425592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8" event={"ID":"fd307ca1-bba4-499a-90e2-d82da526f489","Type":"ContainerDied","Data":"44c454f31333d4c266f1ccf88a26db2dc0511cb09c8ce7079c5edcef1533172a"}
Sep 09 03:23:10.491670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.491607    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:23:10.558949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.558910    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd307ca1-bba4-499a-90e2-d82da526f489-tls-certs\") pod \"fd307ca1-bba4-499a-90e2-d82da526f489\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") "
Sep 09 03:23:10.559156 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.558969    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-home\") pod \"fd307ca1-bba4-499a-90e2-d82da526f489\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") "
Sep 09 03:23:10.559156 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.558997    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-dshm\") pod \"fd307ca1-bba4-499a-90e2-d82da526f489\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") "
Sep 09 03:23:10.559156 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.559057    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-tmp-dir\") pod \"fd307ca1-bba4-499a-90e2-d82da526f489\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") "
Sep 09 03:23:10.559156 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.559112    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkwc8\" (UniqueName: \"kubernetes.io/projected/fd307ca1-bba4-499a-90e2-d82da526f489-kube-api-access-bkwc8\") pod \"fd307ca1-bba4-499a-90e2-d82da526f489\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") "
Sep 09 03:23:10.559392 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.559162    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-model-cache\") pod \"fd307ca1-bba4-499a-90e2-d82da526f489\" (UID: \"fd307ca1-bba4-499a-90e2-d82da526f489\") "
Sep 09 03:23:10.559392 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.559316    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-home" (OuterVolumeSpecName: "home") pod "fd307ca1-bba4-499a-90e2-d82da526f489" (UID: "fd307ca1-bba4-499a-90e2-d82da526f489"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:10.559392 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.559337    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd307ca1-bba4-499a-90e2-d82da526f489" (UID: "fd307ca1-bba4-499a-90e2-d82da526f489"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:10.559532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.559436    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd307ca1-bba4-499a-90e2-d82da526f489" (UID: "fd307ca1-bba4-499a-90e2-d82da526f489"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:10.559532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.559505    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:23:10.559702 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.559581    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:23:10.561606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.561571    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd307ca1-bba4-499a-90e2-d82da526f489-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd307ca1-bba4-499a-90e2-d82da526f489" (UID: "fd307ca1-bba4-499a-90e2-d82da526f489"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:23:10.561732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.561608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-dshm" (OuterVolumeSpecName: "dshm") pod "fd307ca1-bba4-499a-90e2-d82da526f489" (UID: "fd307ca1-bba4-499a-90e2-d82da526f489"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:10.561732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.561643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd307ca1-bba4-499a-90e2-d82da526f489-kube-api-access-bkwc8" (OuterVolumeSpecName: "kube-api-access-bkwc8") pod "fd307ca1-bba4-499a-90e2-d82da526f489" (UID: "fd307ca1-bba4-499a-90e2-d82da526f489"). InnerVolumeSpecName "kube-api-access-bkwc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:23:10.660029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.659987    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkwc8\" (UniqueName: \"kubernetes.io/projected/fd307ca1-bba4-499a-90e2-d82da526f489-kube-api-access-bkwc8\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:23:10.660029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.660023    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:23:10.660029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.660032    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd307ca1-bba4-499a-90e2-d82da526f489-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:23:10.660029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:10.660042    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd307ca1-bba4-499a-90e2-d82da526f489-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:23:11.433507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:11.433475    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"
Sep 09 03:23:11.434046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:11.433517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8" event={"ID":"fd307ca1-bba4-499a-90e2-d82da526f489","Type":"ContainerDied","Data":"87def3f2dfe52d53768e7c627b25ad0340da6a41fdc117b30b45eacb7a901bf2"}
Sep 09 03:23:11.434046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:11.433590    2562 scope.go:117] "RemoveContainer" containerID="44c454f31333d4c266f1ccf88a26db2dc0511cb09c8ce7079c5edcef1533172a"
Sep 09 03:23:11.455208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:11.455174    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"]
Sep 09 03:23:11.459102 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:11.459075    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-f99e31e6-kserve-85d4d599ff-d56q8"]
Sep 09 03:23:11.938262 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:11.938229    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd307ca1-bba4-499a-90e2-d82da526f489" path="/var/lib/kubelet/pods/fd307ca1-bba4-499a-90e2-d82da526f489/volumes"
Sep 09 03:23:13.717650 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:13.717606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:15.943959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:15.943921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:21.406993 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:21.406955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:21.649197 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:21.649160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:22.086709 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.086678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:22.351310 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.351218    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"]
Sep 09 03:23:22.352235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352204    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy"
Sep 09 03:23:22.352715 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352688    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy"
Sep 09 03:23:22.352870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352749    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd307ca1-bba4-499a-90e2-d82da526f489" containerName="main"
Sep 09 03:23:22.352870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352760    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd307ca1-bba4-499a-90e2-d82da526f489" containerName="main"
Sep 09 03:23:22.352870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352774    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy"
Sep 09 03:23:22.352870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352783    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy"
Sep 09 03:23:22.353200 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352926    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd307ca1-bba4-499a-90e2-d82da526f489" containerName="main"
Sep 09 03:23:22.353200 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352946    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy"
Sep 09 03:23:22.353200 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.352958    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="19f119c2-6eac-4f11-b941-ebe4b44dcd37" containerName="istio-proxy"
Sep 09 03:23:22.356135 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.356111    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.362839 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.362813    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-4ce4c1-openshift-default-dockercfg-2gln5\""
Sep 09 03:23:22.373799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.373498    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"]
Sep 09 03:23:22.388498 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.388444    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:22.475396 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-credential-socket\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.475857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475406    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsn79\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-kube-api-access-tsn79\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.475857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-data\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.475857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istiod-ca-cert\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.475857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-envoy\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.475857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-podinfo\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.475857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-token\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.475857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-certs\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.475857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.475769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-socket\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.576873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.576834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-envoy\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.576873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.576884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-podinfo\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577159 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.576953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-token\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577230 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.577156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-certs\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577295 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.577242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-socket\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.577404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-credential-socket\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577557 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.577454    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsn79\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-kube-api-access-tsn79\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577557 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.577522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-data\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.577589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istiod-ca-cert\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.577642    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-certs\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.577875 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.577803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-socket\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.578035 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.578009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-credential-socket\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.578115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.578066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-data\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.578713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.578683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istiod-ca-cert\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.579951 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.579926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-envoy\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.580401 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.580380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-podinfo\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.585454 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.585427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-token\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.585778 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.585759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsn79\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-kube-api-access-tsn79\") pod \"e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") " pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.668259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.668160    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:22.721803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.721751    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:22.949596 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.949566    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"]
Sep 09 03:23:22.951039 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:23:22.951010    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e2f13b_fd37_4a78_96e0_f65504370ee3.slice/crio-6e331bf8985f56c948e3df766e7b5440412d869cf3ffbbd2ce065ea1aed9590b WatchSource:0}: Error finding container 6e331bf8985f56c948e3df766e7b5440412d869cf3ffbbd2ce065ea1aed9590b: Status 404 returned error can't find the container with id 6e331bf8985f56c948e3df766e7b5440412d869cf3ffbbd2ce065ea1aed9590b
Sep 09 03:23:22.953436 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.953398    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:23:22.953533 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.953472    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:23:22.953533 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:22.953499    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:23:23.513968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:23.513929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" event={"ID":"c1e2f13b-fd37-4a78-96e0-f65504370ee3","Type":"ContainerStarted","Data":"bb97e299dd6ee93001846993d7045242e3144b84375602a13e5065c1eed22f60"}
Sep 09 03:23:23.514390 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:23.513997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" event={"ID":"c1e2f13b-fd37-4a78-96e0-f65504370ee3","Type":"ContainerStarted","Data":"6e331bf8985f56c948e3df766e7b5440412d869cf3ffbbd2ce065ea1aed9590b"}
Sep 09 03:23:23.537809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:23.537749    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" podStartSLOduration=1.537732305 podStartE2EDuration="1.537732305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:23:23.534819022 +0000 UTC m=+1052.097145566" watchObservedRunningTime="2026-09-09 03:23:23.537732305 +0000 UTC m=+1052.100058902"
Sep 09 03:23:23.669272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:23.669240    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:24.674592 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:24.674555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:25.439823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:25.439770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:25.526469 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:25.526439    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:25.527417 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:25.527400    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:25.531600 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:25.531568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:26.739059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.739025    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"]
Sep 09 03:23:26.741891 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.741864    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.746702 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.746234    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4ce4c1-serving-cert\""
Sep 09 03:23:26.757198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.757171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"]
Sep 09 03:23:26.782888 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.782850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:26.822916 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.822873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-metrics-certs\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.822916 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.822922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.823157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.823038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdth6\" (UniqueName: \"kubernetes.io/projected/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-kube-api-access-mdth6\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.823200 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.823151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-openshift-service-ca\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.860041 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.859998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:26.924568 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.924493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-metrics-certs\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.925369 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.925342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.925757 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:23:26.925634    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-serving-cert: secret "maas-api-e2e-explicit-4ce4c1-serving-cert" not found
Sep 09 03:23:26.925996 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.925753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdth6\" (UniqueName: \"kubernetes.io/projected/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-kube-api-access-mdth6\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.926404 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:23:26.926387    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls podName:7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87 nodeName:}" failed. No retries permitted until 2026-09-09 03:23:27.426362565 +0000 UTC m=+1055.988689102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls") pod "maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq" (UID: "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87") : secret "maas-api-e2e-explicit-4ce4c1-serving-cert" not found
Sep 09 03:23:26.926734 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.926715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-openshift-service-ca\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.928203 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.928185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-openshift-service-ca\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.930945 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.929085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-metrics-certs\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:26.936264 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:26.936228    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdth6\" (UniqueName: \"kubernetes.io/projected/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-kube-api-access-mdth6\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:27.069687 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:27.069646    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:27.406481 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:27.406384    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:27.431306 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:27.431274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:27.434136 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:27.434110    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls\") pod \"maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:27.505828 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:27.505789    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:27.659504 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:27.659387    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:27.960333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:27.960292    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"]
Sep 09 03:23:27.962673 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:23:27.962645    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbb00d3_36f9_4dfc_b6d2_2fbbda655d87.slice/crio-4d8da68dacb7e951a35b7e8da48fec6e86aa785c94fd5249df3bbc761306de4f WatchSource:0}: Error finding container 4d8da68dacb7e951a35b7e8da48fec6e86aa785c94fd5249df3bbc761306de4f: Status 404 returned error can't find the container with id 4d8da68dacb7e951a35b7e8da48fec6e86aa785c94fd5249df3bbc761306de4f
Sep 09 03:23:27.964170 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:27.964150    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:23:28.548766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:28.548730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq" event={"ID":"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87","Type":"ContainerStarted","Data":"9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817"}
Sep 09 03:23:28.548766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:28.548785    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq" event={"ID":"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87","Type":"ContainerStarted","Data":"4d8da68dacb7e951a35b7e8da48fec6e86aa785c94fd5249df3bbc761306de4f"}
Sep 09 03:23:28.549044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:28.548860    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:28.581335 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:28.581270    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq" podStartSLOduration=2.159876314 podStartE2EDuration="2.581248281s" totalImagesPullingTime="421.371967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:23:28.578624991 +0000 UTC m=+1057.140951537" watchObservedRunningTime="2026-09-09 03:23:28.581248281 +0000 UTC m=+1057.143574826"
Sep 09 03:23:32.859752 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:32.859718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:33.004696 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:33.004661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:34.560902 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:34.560873    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:23:39.320820 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:39.320778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:39.407663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:39.407621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:39.504360 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:39.504324    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:44.665518 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:44.665416    2562 generic.go:356] "Generic (PLEG): container finished" podID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerID="bb97e299dd6ee93001846993d7045242e3144b84375602a13e5065c1eed22f60" exitCode=0
Sep 09 03:23:44.665518 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:44.665498    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" event={"ID":"c1e2f13b-fd37-4a78-96e0-f65504370ee3","Type":"ContainerDied","Data":"bb97e299dd6ee93001846993d7045242e3144b84375602a13e5065c1eed22f60"}
Sep 09 03:23:44.666140 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:44.666040    2562 scope.go:117] "RemoveContainer" containerID="bb97e299dd6ee93001846993d7045242e3144b84375602a13e5065c1eed22f60"
Sep 09 03:23:44.667195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:44.667166    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:23:44.667278 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:44.667236    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:23:44.667278 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:44.667266    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:23:45.673372 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:45.673334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" event={"ID":"c1e2f13b-fd37-4a78-96e0-f65504370ee3","Type":"ContainerStarted","Data":"a80f77d9a08a722a09b96894b8b451bdac960f602ff1b5547fbfa3ba44e184ee"}
Sep 09 03:23:46.669334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:46.669299    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:46.669654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:46.669356    2562 patch_prober.go:28] interesting pod/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:23:46.669840 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:46.669807    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 03:23:47.669780 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:47.669290    2562 patch_prober.go:28] interesting pod/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:23:47.669780 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:47.669351    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 03:23:48.669347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:48.669303    2562 patch_prober.go:28] interesting pod/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:23:48.669666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:48.669635    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 03:23:49.668989 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:49.668933    2562 patch_prober.go:28] interesting pod/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:23:49.669615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:49.669585    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 03:23:50.347978 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:50.347940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:50.669350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:50.669251    2562 patch_prober.go:28] interesting pod/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:23:50.669350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:50.669321    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 03:23:51.753606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:51.753570    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:51.754119 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:51.753971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:51.754832 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:51.754811    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:23:55.334960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:23:55.334922    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:01.084263 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:01.084216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:01.188386 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:01.188354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:01.284706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:01.284670    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:04.140899 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:04.140867    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:11.768639 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:11.768600    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:12.526106 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:12.526066    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:18.297666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:18.297624    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:18.367703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:18.367662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:18.451657 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:18.451604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:18.550973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:18.550858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:18.651132 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:18.651095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:19.207736 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:19.207706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:19.306854 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:19.306822    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:19.404034 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:19.403997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:20.793003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:20.792967    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:26.529616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:26.529574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:26.639521 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:26.639483    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:26.732266 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:26.732217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:27.218193 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:27.218158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:33.174051 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:33.174014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:33.239172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:33.239133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:33.388311 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:33.388278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:33.767470 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:33.767431    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:33.937660 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:33.937620    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:36.961509 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:36.961465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:37.062843 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.062798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:37.280516 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.280416    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"]
Sep 09 03:24:37.280755 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.280724    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq" podUID="7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" containerName="maas-api" containerID="cri-o://9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817" gracePeriod=30
Sep 09 03:24:37.314805 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.314774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:37.413429 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.413385    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:37.498100 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.498041    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:37.548172 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.548146    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:24:37.556789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.556758    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:37.677792 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.677762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-openshift-service-ca\") pod \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") "
Sep 09 03:24:37.678007 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.677854    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdth6\" (UniqueName: \"kubernetes.io/projected/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-kube-api-access-mdth6\") pod \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") "
Sep 09 03:24:37.678007 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.677953    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls\") pod \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") "
Sep 09 03:24:37.678137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.678057    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-metrics-certs\") pod \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\" (UID: \"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87\") "
Sep 09 03:24:37.678228 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.678207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" (UID: "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:37.678414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.678396    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:37.680262 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.680229    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" (UID: "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:24:37.680262 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.680244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-kube-api-access-mdth6" (OuterVolumeSpecName: "kube-api-access-mdth6") pod "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" (UID: "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87"). InnerVolumeSpecName "kube-api-access-mdth6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:24:37.680413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.680297    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" (UID: "7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:24:37.779948 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.779902    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdth6\" (UniqueName: \"kubernetes.io/projected/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-kube-api-access-mdth6\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:37.779948 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.779940    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:37.779948 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.779950    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:37.843701 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.843663    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:37.994369 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.994330    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" containerID="9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817" exitCode=0
Sep 09 03:24:37.994851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.994385    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"
Sep 09 03:24:37.994851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.994412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq" event={"ID":"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87","Type":"ContainerDied","Data":"9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817"}
Sep 09 03:24:37.994851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.994478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq" event={"ID":"7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87","Type":"ContainerDied","Data":"4d8da68dacb7e951a35b7e8da48fec6e86aa785c94fd5249df3bbc761306de4f"}
Sep 09 03:24:37.994851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:37.994507    2562 scope.go:117] "RemoveContainer" containerID="9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817"
Sep 09 03:24:38.005139 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.005118    2562 scope.go:117] "RemoveContainer" containerID="9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817"
Sep 09 03:24:38.005441 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:24:38.005422    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817\": container with ID starting with 9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817 not found: ID does not exist" containerID="9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817"
Sep 09 03:24:38.005491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.005458    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817"} err="failed to get container status \"9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817\": rpc error: code = NotFound desc = could not find container \"9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817\": container with ID starting with 9914b121d0f8d4200b0df132225c68a9049807fa3a30ff6094badec0ff6eb817 not found: ID does not exist"
Sep 09 03:24:38.015915 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.015884    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"]
Sep 09 03:24:38.020759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.020734    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ce4c1-577cdb7c75-dbndq"]
Sep 09 03:24:38.234914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.234812    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"]
Sep 09 03:24:38.235472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.235456    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" containerName="maas-api"
Sep 09 03:24:38.235563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.235474    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" containerName="maas-api"
Sep 09 03:24:38.235630 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.235603    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" containerName="maas-api"
Sep 09 03:24:38.238625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.238599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.242260 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.242226    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-57aecdff-serving-cert\""
Sep 09 03:24:38.249395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.249363    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"]
Sep 09 03:24:38.272069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.272032    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:38.336260 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.336214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:38.386996 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.386950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ssw\" (UniqueName: \"kubernetes.io/projected/f652b3e4-8a03-47ff-a076-0f4791125bc8-kube-api-access-t9ssw\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.387195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.387018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-metrics-certs\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.387195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.387115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f652b3e4-8a03-47ff-a076-0f4791125bc8-openshift-service-ca\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.387195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.387170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-maas-api-tls\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.488718 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.488621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f652b3e4-8a03-47ff-a076-0f4791125bc8-openshift-service-ca\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.488718 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.488693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-maas-api-tls\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.488960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.488781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ssw\" (UniqueName: \"kubernetes.io/projected/f652b3e4-8a03-47ff-a076-0f4791125bc8-kube-api-access-t9ssw\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.488960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.488831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-metrics-certs\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.489433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.489409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f652b3e4-8a03-47ff-a076-0f4791125bc8-openshift-service-ca\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.491390 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.491368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-metrics-certs\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.491487 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.491426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-maas-api-tls\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.497872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.497849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ssw\" (UniqueName: \"kubernetes.io/projected/f652b3e4-8a03-47ff-a076-0f4791125bc8-kube-api-access-t9ssw\") pod \"maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.519623 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.519588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:38.552661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.552618    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:38.833786 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.833760    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"]
Sep 09 03:24:38.834959 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:24:38.834927    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf652b3e4_8a03_47ff_a076_0f4791125bc8.slice/crio-576381aea22407527aa89ca27d8f64756d84a14553dedfce1737d5cd5109a2f6 WatchSource:0}: Error finding container 576381aea22407527aa89ca27d8f64756d84a14553dedfce1737d5cd5109a2f6: Status 404 returned error can't find the container with id 576381aea22407527aa89ca27d8f64756d84a14553dedfce1737d5cd5109a2f6
Sep 09 03:24:38.850770 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.850738    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:38.950230 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:38.950198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:39.002893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:39.002860    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np" event={"ID":"f652b3e4-8a03-47ff-a076-0f4791125bc8","Type":"ContainerStarted","Data":"576381aea22407527aa89ca27d8f64756d84a14553dedfce1737d5cd5109a2f6"}
Sep 09 03:24:39.940010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:39.939973    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87" path="/var/lib/kubelet/pods/7fbb00d3-36f9-4dfc-b6d2-2fbbda655d87/volumes"
Sep 09 03:24:40.019790 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:40.019753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np" event={"ID":"f652b3e4-8a03-47ff-a076-0f4791125bc8","Type":"ContainerStarted","Data":"8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45"}
Sep 09 03:24:40.020238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:40.019970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:40.037174 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:40.037117    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np" podStartSLOduration=1.6425974970000001 podStartE2EDuration="2.037098963s" totalImagesPullingTime="394.501466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:40.034427817 +0000 UTC m=+1128.596754363" watchObservedRunningTime="2026-09-09 03:24:40.037098963 +0000 UTC m=+1128.599425591"
Sep 09 03:24:43.684162 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:43.684095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:43.787169 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:43.787129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:43.924828 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:43.924783    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"]
Sep 09 03:24:43.925050 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:43.925025    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy" containerID="cri-o://a80f77d9a08a722a09b96894b8b451bdac960f602ff1b5547fbfa3ba44e184ee" gracePeriod=30
Sep 09 03:24:43.948375 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:43.948243    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:44.376602 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.376560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:44.635078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.634973    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"]
Sep 09 03:24:44.638246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.638221    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.640560 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.640515    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-05aade-openshift-default-dockercfg-8nsx2\""
Sep 09 03:24:44.662416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.662355    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"]
Sep 09 03:24:44.671736 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.671692    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-socket\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-certs\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-envoy\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-credential-socket\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-data\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758686    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-token\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-podinfo\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758793    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrr9\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-kube-api-access-dsrr9\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.761083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.758842    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istiod-ca-cert\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.859639 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.859596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-credential-socket\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.859853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.859649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-data\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.859853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.859680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-token\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.859853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.859734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-podinfo\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.859853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.859765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrr9\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-kube-api-access-dsrr9\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.859999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istiod-ca-cert\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.860087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-data\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.860087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-credential-socket\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.860114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-socket\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.860155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-certs\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.860210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-envoy\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860455 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.860433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-socket\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860534 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.860508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-certs\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.860787 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.860662    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istiod-ca-cert\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.862486 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.862461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-envoy\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.862613 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.862584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-podinfo\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.867467 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.867442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-token\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.867593 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.867454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrr9\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-kube-api-access-dsrr9\") pod \"e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") " pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:44.963574 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:44.963460    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:45.246949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:45.245923    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"]
Sep 09 03:24:45.248195 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:24:45.248163    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2a8bd6_2ebd_4ece_b8b8_1749be61dfe0.slice/crio-59ff623b1d5c247944349933913d0e98d17245bcc2013247870c12dd09baf834 WatchSource:0}: Error finding container 59ff623b1d5c247944349933913d0e98d17245bcc2013247870c12dd09baf834: Status 404 returned error can't find the container with id 59ff623b1d5c247944349933913d0e98d17245bcc2013247870c12dd09baf834
Sep 09 03:24:45.251439 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:45.251394    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:24:45.251567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:45.251505    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:24:45.251613 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:45.251581    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:24:45.423412 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:45.423338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:46.032346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.032316    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:24:46.064947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.064895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2" event={"ID":"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0","Type":"ContainerStarted","Data":"a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b"}
Sep 09 03:24:46.065157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.064963    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2" event={"ID":"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0","Type":"ContainerStarted","Data":"59ff623b1d5c247944349933913d0e98d17245bcc2013247870c12dd09baf834"}
Sep 09 03:24:46.087277 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.087211    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2" podStartSLOduration=2.087191763 podStartE2EDuration="2.087191763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:46.085048494 +0000 UTC m=+1134.647375043" watchObservedRunningTime="2026-09-09 03:24:46.087191763 +0000 UTC m=+1134.649518308"
Sep 09 03:24:46.355885 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.355842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:46.448764 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.448719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:46.592198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.592166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:46.964479 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.964445    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:46.969765 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:46.969736    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:47.071882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:47.071847    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:47.073286 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:47.073268    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:24:49.087060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.087025    2562 generic.go:356] "Generic (PLEG): container finished" podID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerID="a80f77d9a08a722a09b96894b8b451bdac960f602ff1b5547fbfa3ba44e184ee" exitCode=0
Sep 09 03:24:49.087532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.087097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" event={"ID":"c1e2f13b-fd37-4a78-96e0-f65504370ee3","Type":"ContainerDied","Data":"a80f77d9a08a722a09b96894b8b451bdac960f602ff1b5547fbfa3ba44e184ee"}
Sep 09 03:24:49.087532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.087172    2562 scope.go:117] "RemoveContainer" containerID="bb97e299dd6ee93001846993d7045242e3144b84375602a13e5065c1eed22f60"
Sep 09 03:24:49.186760 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.186735    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:24:49.304709 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.304677    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsn79\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-kube-api-access-tsn79\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.304927 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.304726    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istiod-ca-cert\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.304927 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.304784    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-podinfo\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.304927 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.304836    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-data\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.305178 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.304964    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-credential-socket\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.305178 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305004    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-socket\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.305178 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-envoy\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.305178 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305070    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-certs\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.305178 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305100    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-token\") pod \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\" (UID: \"c1e2f13b-fd37-4a78-96e0-f65504370ee3\") "
Sep 09 03:24:49.305178 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305109    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:49.305491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:49.305491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305262    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:49.305491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305391    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:49.305491 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305436    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:49.305942 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305656    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:49.305942 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305677    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:49.305942 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305692    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:49.305942 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305706    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:49.305942 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.305720    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:49.307276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.307247    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-kube-api-access-tsn79" (OuterVolumeSpecName: "kube-api-access-tsn79") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "kube-api-access-tsn79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:24:49.307499 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.307475    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:24:49.307810 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.307791    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:49.307868 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.307811    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1e2f13b-fd37-4a78-96e0-f65504370ee3" (UID: "c1e2f13b-fd37-4a78-96e0-f65504370ee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:24:49.406681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.406645    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:49.406681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.406675    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:49.406681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.406683    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsn79\" (UniqueName: \"kubernetes.io/projected/c1e2f13b-fd37-4a78-96e0-f65504370ee3-kube-api-access-tsn79\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:49.406935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:49.406696    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2f13b-fd37-4a78-96e0-f65504370ee3-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:24:50.095288 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:50.095252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg" event={"ID":"c1e2f13b-fd37-4a78-96e0-f65504370ee3","Type":"ContainerDied","Data":"6e331bf8985f56c948e3df766e7b5440412d869cf3ffbbd2ce065ea1aed9590b"}
Sep 09 03:24:50.095855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:50.095306    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"
Sep 09 03:24:50.095855 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:50.095311    2562 scope.go:117] "RemoveContainer" containerID="a80f77d9a08a722a09b96894b8b451bdac960f602ff1b5547fbfa3ba44e184ee"
Sep 09 03:24:50.112034 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:50.111996    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"]
Sep 09 03:24:50.116655 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:50.116629    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-4ce4c1-openshift-default-75dd7b87d8-lvbjg"]
Sep 09 03:24:51.940783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:24:51.940752    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" path="/var/lib/kubelet/pods/c1e2f13b-fd37-4a78-96e0-f65504370ee3/volumes"
Sep 09 03:25:01.917313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:01.917261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:02.588580 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:02.588522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:02.684954 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:02.684918    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:03.024269 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:03.024159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:08.197969 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.197922    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"]
Sep 09 03:25:08.198683 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.198660    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy"
Sep 09 03:25:08.198784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.198686    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy"
Sep 09 03:25:08.198784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.198711    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy"
Sep 09 03:25:08.198784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.198719    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy"
Sep 09 03:25:08.198940 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.198856    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy"
Sep 09 03:25:08.204158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.204133    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.207195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.207160    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-05aade\"/\"default-dockercfg-rqgp6\""
Sep 09 03:25:08.207195 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.207192    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-05aade\"/\"openshift-service-ca.crt\""
Sep 09 03:25:08.207406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.207192    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-05aade\"/\"kube-root-ca.crt\""
Sep 09 03:25:08.207406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.207165    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-05aade\"/\"noten-model-05aade-kserve-self-signed-certs\""
Sep 09 03:25:08.212159 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.212129    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"]
Sep 09 03:25:08.231508 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.231478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:08.289665 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.289635    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st9k9\" (UniqueName: \"kubernetes.io/projected/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-kube-api-access-st9k9\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.289818 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.289720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tmp-dir\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.289818 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.289774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-model-cache\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.289947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.289902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-home\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.289947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.289934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tls-certs\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.290026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.289957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-dshm\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391014 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.390972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-home\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391014 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.391020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tls-certs\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.391049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-dshm\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.391077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st9k9\" (UniqueName: \"kubernetes.io/projected/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-kube-api-access-st9k9\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.391114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tmp-dir\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391292 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.391172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-model-cache\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391505 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.391452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-home\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391593 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.391573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-model-cache\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.391664 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.391641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tmp-dir\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.393692 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.393671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-dshm\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.393877 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.393860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tls-certs\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.400088 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.400034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st9k9\" (UniqueName: \"kubernetes.io/projected/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-kube-api-access-st9k9\") pod \"noten-model-05aade-kserve-dbbdb9f5-8gr9h\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") " pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.517184 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.517089    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:08.790756 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.790727    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"]
Sep 09 03:25:08.791336 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:25:08.791307    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0a9f8cf_e5ab_4aaa_9499_fe810cf29281.slice/crio-abf7303ea7579148130b0118e44f353981fdee874ddb33344680b6e60a8af817 WatchSource:0}: Error finding container abf7303ea7579148130b0118e44f353981fdee874ddb33344680b6e60a8af817: Status 404 returned error can't find the container with id abf7303ea7579148130b0118e44f353981fdee874ddb33344680b6e60a8af817
Sep 09 03:25:08.880959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.880927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:08.943217 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:08.943175    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:09.213274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:09.213185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h" event={"ID":"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281","Type":"ContainerStarted","Data":"e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427"}
Sep 09 03:25:09.213274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:09.213226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h" event={"ID":"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281","Type":"ContainerStarted","Data":"abf7303ea7579148130b0118e44f353981fdee874ddb33344680b6e60a8af817"}
Sep 09 03:25:09.230766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:09.230716    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h" podStartSLOduration=1.230699136 podStartE2EDuration="1.230699136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:09.228756187 +0000 UTC m=+1157.791082743" watchObservedRunningTime="2026-09-09 03:25:09.230699136 +0000 UTC m=+1157.793025681"
Sep 09 03:25:10.587201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:10.587169    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"]
Sep 09 03:25:10.606087 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:10.606035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:16.867756 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:16.867656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:17.109990 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:17.109948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:17.780723 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:17.780690    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"]
Sep 09 03:25:17.807123 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:17.807091    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:17.895946 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:17.895911    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"]
Sep 09 03:25:17.896381 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:17.896123    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2" podUID="8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" containerName="istio-proxy" containerID="cri-o://a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b" gracePeriod=30
Sep 09 03:25:17.920713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:17.920678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:18.135048 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:18.135012    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:18.311364 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:18.311326    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:18.517835 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:18.517737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:19.620922 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:19.620887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:19.908496 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:19.908395    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"]
Sep 09 03:25:19.908708 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:19.908684    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np" podUID="f652b3e4-8a03-47ff-a076-0f4791125bc8" containerName="maas-api" containerID="cri-o://8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45" gracePeriod=30
Sep 09 03:25:19.947529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:19.947498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:20.019818 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.019778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:20.116717 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.116683    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:20.162730 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.162670    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:25:20.216256 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.216217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:20.285041 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.285009    2562 generic.go:356] "Generic (PLEG): container finished" podID="f652b3e4-8a03-47ff-a076-0f4791125bc8" containerID="8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45" exitCode=0
Sep 09 03:25:20.285255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.285062    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"
Sep 09 03:25:20.285255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.285100    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np" event={"ID":"f652b3e4-8a03-47ff-a076-0f4791125bc8","Type":"ContainerDied","Data":"8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45"}
Sep 09 03:25:20.285255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.285189    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np" event={"ID":"f652b3e4-8a03-47ff-a076-0f4791125bc8","Type":"ContainerDied","Data":"576381aea22407527aa89ca27d8f64756d84a14553dedfce1737d5cd5109a2f6"}
Sep 09 03:25:20.285255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.285212    2562 scope.go:117] "RemoveContainer" containerID="8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45"
Sep 09 03:25:20.298370 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.298343    2562 scope.go:117] "RemoveContainer" containerID="8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45"
Sep 09 03:25:20.298939 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:25:20.298908    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45\": container with ID starting with 8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45 not found: ID does not exist" containerID="8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45"
Sep 09 03:25:20.299083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.298948    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45"} err="failed to get container status \"8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45\": rpc error: code = NotFound desc = could not find container \"8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45\": container with ID starting with 8e85991d4d2bc8e0221926ef5205bef1960b8073b2fab8b92dbd4508273f5a45 not found: ID does not exist"
Sep 09 03:25:20.314334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.314304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f652b3e4-8a03-47ff-a076-0f4791125bc8-openshift-service-ca\") pod \"f652b3e4-8a03-47ff-a076-0f4791125bc8\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") "
Sep 09 03:25:20.314535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.314346    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-maas-api-tls\") pod \"f652b3e4-8a03-47ff-a076-0f4791125bc8\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") "
Sep 09 03:25:20.314535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.314380    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9ssw\" (UniqueName: \"kubernetes.io/projected/f652b3e4-8a03-47ff-a076-0f4791125bc8-kube-api-access-t9ssw\") pod \"f652b3e4-8a03-47ff-a076-0f4791125bc8\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") "
Sep 09 03:25:20.314535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.314447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-metrics-certs\") pod \"f652b3e4-8a03-47ff-a076-0f4791125bc8\" (UID: \"f652b3e4-8a03-47ff-a076-0f4791125bc8\") "
Sep 09 03:25:20.314832 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.314792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f652b3e4-8a03-47ff-a076-0f4791125bc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f652b3e4-8a03-47ff-a076-0f4791125bc8" (UID: "f652b3e4-8a03-47ff-a076-0f4791125bc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:20.314949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.314930    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f652b3e4-8a03-47ff-a076-0f4791125bc8-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:20.317394 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.317368    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f652b3e4-8a03-47ff-a076-0f4791125bc8" (UID: "f652b3e4-8a03-47ff-a076-0f4791125bc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:20.317507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.317422    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f652b3e4-8a03-47ff-a076-0f4791125bc8-kube-api-access-t9ssw" (OuterVolumeSpecName: "kube-api-access-t9ssw") pod "f652b3e4-8a03-47ff-a076-0f4791125bc8" (UID: "f652b3e4-8a03-47ff-a076-0f4791125bc8"). InnerVolumeSpecName "kube-api-access-t9ssw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:20.317507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.317490    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f652b3e4-8a03-47ff-a076-0f4791125bc8" (UID: "f652b3e4-8a03-47ff-a076-0f4791125bc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:20.415616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.415495    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:20.415616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.415529    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9ssw\" (UniqueName: \"kubernetes.io/projected/f652b3e4-8a03-47ff-a076-0f4791125bc8-kube-api-access-t9ssw\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:20.415616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.415579    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f652b3e4-8a03-47ff-a076-0f4791125bc8-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:20.482403 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.482368    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:20.609896 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.609867    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"]
Sep 09 03:25:20.614323 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:20.614293    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57aecdff-6d9775f79c-lz4np"]
Sep 09 03:25:21.938433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:21.938402    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f652b3e4-8a03-47ff-a076-0f4791125bc8" path="/var/lib/kubelet/pods/f652b3e4-8a03-47ff-a076-0f4791125bc8/volumes"
Sep 09 03:25:23.152645 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.152619    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:25:23.244793 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.244716    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsrr9\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-kube-api-access-dsrr9\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.244793 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.244770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istiod-ca-cert\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.244793 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.244804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-data\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.245074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.244825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-envoy\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.245074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.244881    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-certs\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.245074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.244910    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-socket\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.245074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.244994    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-token\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.245074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245024    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-credential-socket\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.245074 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245048    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-podinfo\") pod \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\" (UID: \"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0\") "
Sep 09 03:25:23.245386 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245196    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:23.245386 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245245    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:23.245497 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245443    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:23.245578 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245533    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:23.245657 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245630    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.245657 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245652    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.245770 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245666    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.245770 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.245721    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:23.247436 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.247413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:25:23.247993 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.247969    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-kube-api-access-dsrr9" (OuterVolumeSpecName: "kube-api-access-dsrr9") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "kube-api-access-dsrr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:23.247993 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.247970    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:23.248173 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.247996    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" (UID: "8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:23.261115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.261085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:23.305791 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.305760    2562 generic.go:356] "Generic (PLEG): container finished" podID="8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" containerID="a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b" exitCode=0
Sep 09 03:25:23.305964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.305807    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"
Sep 09 03:25:23.305964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.305859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2" event={"ID":"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0","Type":"ContainerDied","Data":"a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b"}
Sep 09 03:25:23.305964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.305913    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2" event={"ID":"8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0","Type":"ContainerDied","Data":"59ff623b1d5c247944349933913d0e98d17245bcc2013247870c12dd09baf834"}
Sep 09 03:25:23.305964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.305935    2562 scope.go:117] "RemoveContainer" containerID="a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b"
Sep 09 03:25:23.315731 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.315712    2562 scope.go:117] "RemoveContainer" containerID="a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b"
Sep 09 03:25:23.315995 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:25:23.315978    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b\": container with ID starting with a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b not found: ID does not exist" containerID="a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b"
Sep 09 03:25:23.316041 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.316004    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b"} err="failed to get container status \"a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b\": rpc error: code = NotFound desc = could not find container \"a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b\": container with ID starting with a145a973dcb8d176fb4918ad94a823c83041cc76ca50fcbb92d2441e4c78607b not found: ID does not exist"
Sep 09 03:25:23.327752 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.327731    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"]
Sep 09 03:25:23.330521 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.330500    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-05aade-openshift-default-6887df7bcb-qx6z2"]
Sep 09 03:25:23.346270 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.346250    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.346383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.346273    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.346383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.346285    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsrr9\" (UniqueName: \"kubernetes.io/projected/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-kube-api-access-dsrr9\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.346383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.346299    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.346383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.346312    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.346383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.346326    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:23.938349 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:23.938315    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" path="/var/lib/kubelet/pods/8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0/volumes"
Sep 09 03:25:25.403801 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.403759    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:25.608601 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.608516    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69" podUID="8dbb1560-6861-4f17-aa36-394c8bdfeab3" containerName="main" containerID="cri-o://f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1" gracePeriod=45
Sep 09 03:25:25.872774 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.872751    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:25:25.973640 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.973534    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-home\") pod \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") "
Sep 09 03:25:25.973795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.973668    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tls-certs\") pod \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") "
Sep 09 03:25:25.973795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.973698    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-dshm\") pod \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") "
Sep 09 03:25:25.973795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.973726    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tmp-dir\") pod \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") "
Sep 09 03:25:25.973795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.973756    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9lwf\" (UniqueName: \"kubernetes.io/projected/8dbb1560-6861-4f17-aa36-394c8bdfeab3-kube-api-access-j9lwf\") pod \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") "
Sep 09 03:25:25.973795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.973771    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-home" (OuterVolumeSpecName: "home") pod "8dbb1560-6861-4f17-aa36-394c8bdfeab3" (UID: "8dbb1560-6861-4f17-aa36-394c8bdfeab3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:25.973795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.973797    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-model-cache\") pod \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\" (UID: \"8dbb1560-6861-4f17-aa36-394c8bdfeab3\") "
Sep 09 03:25:25.974143 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.973987    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dbb1560-6861-4f17-aa36-394c8bdfeab3" (UID: "8dbb1560-6861-4f17-aa36-394c8bdfeab3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:25.974143 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.974113    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dbb1560-6861-4f17-aa36-394c8bdfeab3" (UID: "8dbb1560-6861-4f17-aa36-394c8bdfeab3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:25.974243 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.974130    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:25.974243 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.974195    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:25.976108 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.976083    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbb1560-6861-4f17-aa36-394c8bdfeab3-kube-api-access-j9lwf" (OuterVolumeSpecName: "kube-api-access-j9lwf") pod "8dbb1560-6861-4f17-aa36-394c8bdfeab3" (UID: "8dbb1560-6861-4f17-aa36-394c8bdfeab3"). InnerVolumeSpecName "kube-api-access-j9lwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:25.976501 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.976480    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dbb1560-6861-4f17-aa36-394c8bdfeab3" (UID: "8dbb1560-6861-4f17-aa36-394c8bdfeab3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:25.976582 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:25.976484    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-dshm" (OuterVolumeSpecName: "dshm") pod "8dbb1560-6861-4f17-aa36-394c8bdfeab3" (UID: "8dbb1560-6861-4f17-aa36-394c8bdfeab3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:26.075248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.075212    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:26.075248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.075241    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9lwf\" (UniqueName: \"kubernetes.io/projected/8dbb1560-6861-4f17-aa36-394c8bdfeab3-kube-api-access-j9lwf\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:26.075248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.075253    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbb1560-6861-4f17-aa36-394c8bdfeab3-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:26.075486 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.075262    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbb1560-6861-4f17-aa36-394c8bdfeab3-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:26.325237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.325201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:26.326294 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.326274    2562 generic.go:356] "Generic (PLEG): container finished" podID="8dbb1560-6861-4f17-aa36-394c8bdfeab3" containerID="f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1" exitCode=0
Sep 09 03:25:26.326362 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.326322    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"
Sep 09 03:25:26.326404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.326325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69" event={"ID":"8dbb1560-6861-4f17-aa36-394c8bdfeab3","Type":"ContainerDied","Data":"f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1"}
Sep 09 03:25:26.326437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.326417    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69" event={"ID":"8dbb1560-6861-4f17-aa36-394c8bdfeab3","Type":"ContainerDied","Data":"5213a5931fb9b4b6c1e4d5dd5f167a7815cba2832e4d73360deb7249a3678da3"}
Sep 09 03:25:26.326437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.326433    2562 scope.go:117] "RemoveContainer" containerID="f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1"
Sep 09 03:25:26.336045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.336029    2562 scope.go:117] "RemoveContainer" containerID="f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1"
Sep 09 03:25:26.336312 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:25:26.336292    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1\": container with ID starting with f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1 not found: ID does not exist" containerID="f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1"
Sep 09 03:25:26.336359 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.336321    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1"} err="failed to get container status \"f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1\": rpc error: code = NotFound desc = could not find container \"f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1\": container with ID starting with f77460b2a881c570ecbc4d2f9416c5176108d5a7379817952cf8fbb0536332b1 not found: ID does not exist"
Sep 09 03:25:26.347817 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.347780    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"]
Sep 09 03:25:26.351635 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.351594    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b27dde/test-model-b27dde-kserve-55fc87c79c-rbb69"]
Sep 09 03:25:26.424766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.424730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:26.581640 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.581515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:26.715505 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.715470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:26.837778 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.837676    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:26.947775 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:26.947743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:27.039983 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.039947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:27.138583 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.138486    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:27.239287 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.239257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:27.774594 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.774562    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"]
Sep 09 03:25:27.775061 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.774811    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg" podUID="5b4d037d-0f61-4ace-8667-230d796a2a1d" containerName="maas-api" containerID="cri-o://a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d" gracePeriod=30
Sep 09 03:25:27.808500 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.808462    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:27.893005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.892976    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:27.939249 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.939221    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dbb1560-6861-4f17-aa36-394c8bdfeab3" path="/var/lib/kubelet/pods/8dbb1560-6861-4f17-aa36-394c8bdfeab3/volumes"
Sep 09 03:25:27.977634 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:27.977586    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:28.052769 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.052739    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:28.250254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.250233    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:25:28.341013 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.340919    2562 generic.go:356] "Generic (PLEG): container finished" podID="5b4d037d-0f61-4ace-8667-230d796a2a1d" containerID="a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d" exitCode=0
Sep 09 03:25:28.341013 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.340973    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"
Sep 09 03:25:28.341013 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.340996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg" event={"ID":"5b4d037d-0f61-4ace-8667-230d796a2a1d","Type":"ContainerDied","Data":"a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d"}
Sep 09 03:25:28.341297 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.341059    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg" event={"ID":"5b4d037d-0f61-4ace-8667-230d796a2a1d","Type":"ContainerDied","Data":"8aa924252427c1042658b88009fa7d1c558b0e7616f7478140d7ebdd3d360398"}
Sep 09 03:25:28.341297 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.341090    2562 scope.go:117] "RemoveContainer" containerID="a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d"
Sep 09 03:25:28.350673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.350653    2562 scope.go:117] "RemoveContainer" containerID="a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d"
Sep 09 03:25:28.351014 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:25:28.350987    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d\": container with ID starting with a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d not found: ID does not exist" containerID="a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d"
Sep 09 03:25:28.351094 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.351026    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d"} err="failed to get container status \"a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d\": rpc error: code = NotFound desc = could not find container \"a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d\": container with ID starting with a0d7729d328bb3e31a5c4428899ed11a888367fadfb2fa33676b671daa5fb49d not found: ID does not exist"
Sep 09 03:25:28.399417 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.399367    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-maas-api-tls\") pod \"5b4d037d-0f61-4ace-8667-230d796a2a1d\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") "
Sep 09 03:25:28.399417 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.399417    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-metrics-certs\") pod \"5b4d037d-0f61-4ace-8667-230d796a2a1d\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") "
Sep 09 03:25:28.399716 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.399438    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4d037d-0f61-4ace-8667-230d796a2a1d-openshift-service-ca\") pod \"5b4d037d-0f61-4ace-8667-230d796a2a1d\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") "
Sep 09 03:25:28.399716 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.399494    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2qj7\" (UniqueName: \"kubernetes.io/projected/5b4d037d-0f61-4ace-8667-230d796a2a1d-kube-api-access-g2qj7\") pod \"5b4d037d-0f61-4ace-8667-230d796a2a1d\" (UID: \"5b4d037d-0f61-4ace-8667-230d796a2a1d\") "
Sep 09 03:25:28.399854 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.399833    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b4d037d-0f61-4ace-8667-230d796a2a1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b4d037d-0f61-4ace-8667-230d796a2a1d" (UID: "5b4d037d-0f61-4ace-8667-230d796a2a1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:28.401916 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.401891    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b4d037d-0f61-4ace-8667-230d796a2a1d" (UID: "5b4d037d-0f61-4ace-8667-230d796a2a1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:28.401916 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.401903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4d037d-0f61-4ace-8667-230d796a2a1d-kube-api-access-g2qj7" (OuterVolumeSpecName: "kube-api-access-g2qj7") pod "5b4d037d-0f61-4ace-8667-230d796a2a1d" (UID: "5b4d037d-0f61-4ace-8667-230d796a2a1d"). InnerVolumeSpecName "kube-api-access-g2qj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:28.402059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.401965    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b4d037d-0f61-4ace-8667-230d796a2a1d" (UID: "5b4d037d-0f61-4ace-8667-230d796a2a1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:28.446240 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.446201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:28.501003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.500960    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:28.501003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.500997    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4d037d-0f61-4ace-8667-230d796a2a1d-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:28.501003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.501009    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4d037d-0f61-4ace-8667-230d796a2a1d-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:28.501226 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.501022    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2qj7\" (UniqueName: \"kubernetes.io/projected/5b4d037d-0f61-4ace-8667-230d796a2a1d-kube-api-access-g2qj7\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:28.665250 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.665218    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"]
Sep 09 03:25:28.671052 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:28.671027    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b27dde-7f5bbfdc86-bv7hg"]
Sep 09 03:25:29.938802 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:29.938770    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b4d037d-0f61-4ace-8667-230d796a2a1d" path="/var/lib/kubelet/pods/5b4d037d-0f61-4ace-8667-230d796a2a1d/volumes"
Sep 09 03:25:32.802024 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:32.801972    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h" podUID="b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" containerName="main" containerID="cri-o://e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427" gracePeriod=45
Sep 09 03:25:33.052572 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.052481    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:33.058894 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.058853    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:33.149078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149039    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st9k9\" (UniqueName: \"kubernetes.io/projected/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-kube-api-access-st9k9\") pod \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") "
Sep 09 03:25:33.149078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-dshm\") pod \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") "
Sep 09 03:25:33.149329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149142    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tmp-dir\") pod \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") "
Sep 09 03:25:33.149329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149173    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-home\") pod \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") "
Sep 09 03:25:33.149329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149211    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-model-cache\") pod \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") "
Sep 09 03:25:33.149329 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149259    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tls-certs\") pod \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\" (UID: \"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281\") "
Sep 09 03:25:33.149561 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149412    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" (UID: "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:33.149561 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149434    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" (UID: "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:33.149642 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149622    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:33.149689 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149643    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:33.149868 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.149821    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-home" (OuterVolumeSpecName: "home") pod "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" (UID: "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:33.151872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.151847    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" (UID: "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:33.151872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.151854    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-dshm" (OuterVolumeSpecName: "dshm") pod "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" (UID: "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:33.152029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.151854    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-kube-api-access-st9k9" (OuterVolumeSpecName: "kube-api-access-st9k9") pod "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" (UID: "b0a9f8cf-e5ab-4aaa-9499-fe810cf29281"). InnerVolumeSpecName "kube-api-access-st9k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:33.220598 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.220531    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:33.250759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.250719    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:33.250759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.250751    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:33.250759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.250762    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st9k9\" (UniqueName: \"kubernetes.io/projected/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-kube-api-access-st9k9\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:33.250997 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.250772    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:33.301963 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.301923    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:33.375901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.375804    2562 generic.go:356] "Generic (PLEG): container finished" podID="b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" containerID="e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427" exitCode=0
Sep 09 03:25:33.375901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.375844    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h" event={"ID":"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281","Type":"ContainerDied","Data":"e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427"}
Sep 09 03:25:33.375901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.375857    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"
Sep 09 03:25:33.375901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.375874    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h" event={"ID":"b0a9f8cf-e5ab-4aaa-9499-fe810cf29281","Type":"ContainerDied","Data":"abf7303ea7579148130b0118e44f353981fdee874ddb33344680b6e60a8af817"}
Sep 09 03:25:33.375901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.375893    2562 scope.go:117] "RemoveContainer" containerID="e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427"
Sep 09 03:25:33.386522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.386496    2562 scope.go:117] "RemoveContainer" containerID="e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427"
Sep 09 03:25:33.386930 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:25:33.386909    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427\": container with ID starting with e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427 not found: ID does not exist" containerID="e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427"
Sep 09 03:25:33.386989 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.386944    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427"} err="failed to get container status \"e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427\": rpc error: code = NotFound desc = could not find container \"e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427\": container with ID starting with e8cd5085513f4abf7e4a3779b4a8607d798b351eadb5e1b7cb3006cf62df2427 not found: ID does not exist"
Sep 09 03:25:33.401554 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.401509    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"]
Sep 09 03:25:33.410715 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.410677    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-05aade/noten-model-05aade-kserve-dbbdb9f5-8gr9h"]
Sep 09 03:25:33.422113 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.422083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:33.939406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:33.939360    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" path="/var/lib/kubelet/pods/b0a9f8cf-e5ab-4aaa-9499-fe810cf29281/volumes"
Sep 09 03:25:35.510594 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:35.510557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:35.609639 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:35.609603    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:35.767779 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:35.767700    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"]
Sep 09 03:25:35.767956 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:35.767930    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy" containerID="cri-o://13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66" gracePeriod=30
Sep 09 03:25:35.817599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:35.817530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:36.318110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:36.318063    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:39.394374 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:39.394330    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:39.498332 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:39.498292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:40.266375 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.266334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:40.539856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.539822    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"]
Sep 09 03:25:40.540283 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540268    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" containerName="istio-proxy"
Sep 09 03:25:40.540354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540285    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" containerName="istio-proxy"
Sep 09 03:25:40.540354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540297    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dbb1560-6861-4f17-aa36-394c8bdfeab3" containerName="main"
Sep 09 03:25:40.540354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540302    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dbb1560-6861-4f17-aa36-394c8bdfeab3" containerName="main"
Sep 09 03:25:40.540354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540315    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" containerName="main"
Sep 09 03:25:40.540354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540320    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" containerName="main"
Sep 09 03:25:40.540354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540331    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b4d037d-0f61-4ace-8667-230d796a2a1d" containerName="maas-api"
Sep 09 03:25:40.540354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540336    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b4d037d-0f61-4ace-8667-230d796a2a1d" containerName="maas-api"
Sep 09 03:25:40.540354 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540356    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f652b3e4-8a03-47ff-a076-0f4791125bc8" containerName="maas-api"
Sep 09 03:25:40.540663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540362    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f652b3e4-8a03-47ff-a076-0f4791125bc8" containerName="maas-api"
Sep 09 03:25:40.540663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540433    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b4d037d-0f61-4ace-8667-230d796a2a1d" containerName="maas-api"
Sep 09 03:25:40.540663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540443    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1e2f13b-fd37-4a78-96e0-f65504370ee3" containerName="istio-proxy"
Sep 09 03:25:40.540663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540449    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dbb1560-6861-4f17-aa36-394c8bdfeab3" containerName="main"
Sep 09 03:25:40.540663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540454    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f652b3e4-8a03-47ff-a076-0f4791125bc8" containerName="maas-api"
Sep 09 03:25:40.540663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540459    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a9f8cf-e5ab-4aaa-9499-fe810cf29281" containerName="main"
Sep 09 03:25:40.540663 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.540466    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2a8bd6-2ebd-4ece-b8b8-1749be61dfe0" containerName="istio-proxy"
Sep 09 03:25:40.545193 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.545165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.548512 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.548486    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-207ebd-openshift-default-dockercfg-dt47g\""
Sep 09 03:25:40.559393 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.559364    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"]
Sep 09 03:25:40.575049 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.575007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:40.620657 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrkr\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-kube-api-access-8nrkr\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.620891 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-data\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.620891 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-socket\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.620891 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85f576-093f-4bb8-85ed-91636531b445-istio-podinfo\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.620891 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-certs\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.621162 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-envoy\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.621162 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-credential-socket\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.621162 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-istio-token\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.621162 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.620968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85f576-093f-4bb8-85ed-91636531b445-istiod-ca-cert\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.721583 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.721530    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-data\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.721583 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.721588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-socket\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.721871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.721699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85f576-093f-4bb8-85ed-91636531b445-istio-podinfo\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.721871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.721752    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-certs\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.721871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.721828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-envoy\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.721871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.721862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-credential-socket\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.722085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.721899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-istio-token\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.722085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.721930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85f576-093f-4bb8-85ed-91636531b445-istiod-ca-cert\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.722085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.722009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrkr\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-kube-api-access-8nrkr\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.722085 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.722013    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-data\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.722285 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.722152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-certs\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.722409 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.722387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-credential-socket\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.722524 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.722493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-socket\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.722699 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.722683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85f576-093f-4bb8-85ed-91636531b445-istiod-ca-cert\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.724348 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.724314    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-envoy\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.724504 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.724484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85f576-093f-4bb8-85ed-91636531b445-istio-podinfo\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.730833 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.730809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-istio-token\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.730941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.730860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrkr\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-kube-api-access-8nrkr\") pod \"e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") " pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:40.819452 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.819419    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:40.857473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:40.857439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:41.031270 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.031237    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:25:41.125703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125599    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-token\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.125703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw2wn\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-kube-api-access-mw2wn\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.125703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125656    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-socket\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.125703 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125677    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-data\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.126126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-envoy\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.126126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-credential-socket\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.126126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125798    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-certs\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.126126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125829    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c71dda-27bd-430a-8e0b-28a3f277f372-istiod-ca-cert\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.126126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125891    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-podinfo\") pod \"84c71dda-27bd-430a-8e0b-28a3f277f372\" (UID: \"84c71dda-27bd-430a-8e0b-28a3f277f372\") "
Sep 09 03:25:41.126126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.125996    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:41.126126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.126030    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-data" (OuterVolumeSpecName: "istio-data") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:41.127069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.126135    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:41.127069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.126190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:41.127069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.126351    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c71dda-27bd-430a-8e0b-28a3f277f372-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:41.127069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.126361    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.127069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.126441    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.127069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.126604    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.127069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.126618    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.128404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.128376    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:25:41.128404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.128376    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-token" (OuterVolumeSpecName: "istio-token") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:41.128606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.128589    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-kube-api-access-mw2wn" (OuterVolumeSpecName: "kube-api-access-mw2wn") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "kube-api-access-mw2wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:41.128679 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.128666    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "84c71dda-27bd-430a-8e0b-28a3f277f372" (UID: "84c71dda-27bd-430a-8e0b-28a3f277f372"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:41.143488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.143461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"]
Sep 09 03:25:41.146446 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:25:41.146421    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e85f576_093f_4bb8_85ed_91636531b445.slice/crio-474fa0af14de87a7eb4d7eb676d24a5d078d8b2520b11a01013ad4f556f58e24 WatchSource:0}: Error finding container 474fa0af14de87a7eb4d7eb676d24a5d078d8b2520b11a01013ad4f556f58e24: Status 404 returned error can't find the container with id 474fa0af14de87a7eb4d7eb676d24a5d078d8b2520b11a01013ad4f556f58e24
Sep 09 03:25:41.149226 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.149184    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:25:41.149330 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.149265    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:25:41.149330 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.149295    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:25:41.228002 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.227971    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c71dda-27bd-430a-8e0b-28a3f277f372-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.228002 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.227999    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.228002 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.228008    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.228221 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.228016    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw2wn\" (UniqueName: \"kubernetes.io/projected/84c71dda-27bd-430a-8e0b-28a3f277f372-kube-api-access-mw2wn\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.228221 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.228026    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c71dda-27bd-430a-8e0b-28a3f277f372-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:41.427585 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.427479    2562 generic.go:356] "Generic (PLEG): container finished" podID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerID="13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66" exitCode=0
Sep 09 03:25:41.427585 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.427568    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"
Sep 09 03:25:41.427809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.427589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" event={"ID":"84c71dda-27bd-430a-8e0b-28a3f277f372","Type":"ContainerDied","Data":"13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66"}
Sep 09 03:25:41.427809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.427658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp" event={"ID":"84c71dda-27bd-430a-8e0b-28a3f277f372","Type":"ContainerDied","Data":"8831d6ecfd4691acba7ee29ad2bbd9693f99bc338e7166bd0aa9949846a063f1"}
Sep 09 03:25:41.427809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.427693    2562 scope.go:117] "RemoveContainer" containerID="13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66"
Sep 09 03:25:41.429803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.429770    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" event={"ID":"2e85f576-093f-4bb8-85ed-91636531b445","Type":"ContainerStarted","Data":"172e0124e24706435f342348817559733045cedd3cc6112247576d22b0a758bb"}
Sep 09 03:25:41.429931 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.429840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" event={"ID":"2e85f576-093f-4bb8-85ed-91636531b445","Type":"ContainerStarted","Data":"474fa0af14de87a7eb4d7eb676d24a5d078d8b2520b11a01013ad4f556f58e24"}
Sep 09 03:25:41.440719 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.440699    2562 scope.go:117] "RemoveContainer" containerID="e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e"
Sep 09 03:25:41.450379 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.450354    2562 scope.go:117] "RemoveContainer" containerID="13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66"
Sep 09 03:25:41.450757 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:25:41.450729    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66\": container with ID starting with 13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66 not found: ID does not exist" containerID="13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66"
Sep 09 03:25:41.450873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.450768    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66"} err="failed to get container status \"13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66\": rpc error: code = NotFound desc = could not find container \"13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66\": container with ID starting with 13ca9bbd29baeb14b19b185eadf6bacd27996a0157a608a586afa1885ffa7e66 not found: ID does not exist"
Sep 09 03:25:41.450873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.450794    2562 scope.go:117] "RemoveContainer" containerID="e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e"
Sep 09 03:25:41.451056 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:25:41.451036    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e\": container with ID starting with e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e not found: ID does not exist" containerID="e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e"
Sep 09 03:25:41.451120 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.451061    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e"} err="failed to get container status \"e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e\": rpc error: code = NotFound desc = could not find container \"e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e\": container with ID starting with e6f44f78374c442cf682002915cd7a3c3e7312945d1073b70c78b3a71602272e not found: ID does not exist"
Sep 09 03:25:41.454220 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.454167    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podStartSLOduration=1.45415172 podStartE2EDuration="1.45415172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:41.452726888 +0000 UTC m=+1190.015053433" watchObservedRunningTime="2026-09-09 03:25:41.45415172 +0000 UTC m=+1190.016478264"
Sep 09 03:25:41.469366 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.469337    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"]
Sep 09 03:25:41.475579 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.475525    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b27dde-openshift-default-85b7b9949d-bv2vp"]
Sep 09 03:25:41.858075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.858043    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:41.940514 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:41.940472    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" path="/var/lib/kubelet/pods/84c71dda-27bd-430a-8e0b-28a3f277f372/volumes"
Sep 09 03:25:42.864110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:42.864078    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:43.338615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:43.338566    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:43.428065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:43.428021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:43.444151 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:43.444118    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:43.445425 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:43.445397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:25:44.463271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.463238    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"]
Sep 09 03:25:44.463873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.463849    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy"
Sep 09 03:25:44.463873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.463873    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy"
Sep 09 03:25:44.464083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.464004    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy"
Sep 09 03:25:44.469164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.469143    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.471352 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.471330    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-207ebd-serving-cert\""
Sep 09 03:25:44.484633 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.484601    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"]
Sep 09 03:25:44.506715 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.506679    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:44.564260 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.564221    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.564482 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.564333    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.564482 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.564368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.564672 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.564486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2668t\" (UniqueName: \"kubernetes.io/projected/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-kube-api-access-2668t\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.584563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.584505    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:44.665453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.665407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.665453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.665461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2668t\" (UniqueName: \"kubernetes.io/projected/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-kube-api-access-2668t\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.665797 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.665504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.665797 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.665647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.666243 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.666223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.668431 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.668401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.668583 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.668405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.675011 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.674978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2668t\" (UniqueName: \"kubernetes.io/projected/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-kube-api-access-2668t\") pod \"maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.685271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.685235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:44.783599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.783526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:44.995439 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:44.995398    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:45.061776 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:45.061747    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"]
Sep 09 03:25:45.062768 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:25:45.062740    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5d11b3_64e6_4433_8c9b_d2d68401c7aa.slice/crio-45a70d1fb65296b0bf168e1dfee8a9d26b1d9fa4a1a2a942ac4c9dbfb7f4a8a7 WatchSource:0}: Error finding container 45a70d1fb65296b0bf168e1dfee8a9d26b1d9fa4a1a2a942ac4c9dbfb7f4a8a7: Status 404 returned error can't find the container with id 45a70d1fb65296b0bf168e1dfee8a9d26b1d9fa4a1a2a942ac4c9dbfb7f4a8a7
Sep 09 03:25:45.097245 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:45.097208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:45.460497 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:45.460402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9" event={"ID":"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa","Type":"ContainerStarted","Data":"45a70d1fb65296b0bf168e1dfee8a9d26b1d9fa4a1a2a942ac4c9dbfb7f4a8a7"}
Sep 09 03:25:46.354161 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.354061    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"]
Sep 09 03:25:46.354666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.354370    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v" podUID="67d5ed21-4e4a-4287-a01c-0a3bb93e3637" containerName="maas-api" containerID="cri-o://2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f" gracePeriod=30
Sep 09 03:25:46.382302 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.382267    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:46.475606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.475500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9" event={"ID":"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa","Type":"ContainerStarted","Data":"cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43"}
Sep 09 03:25:46.475784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.475624    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:46.483378 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.483348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:46.494245 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.494186    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9" podStartSLOduration=2.013774929 podStartE2EDuration="2.494166125s" totalImagesPullingTime="480.391196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:46.492262476 +0000 UTC m=+1195.054589021" watchObservedRunningTime="2026-09-09 03:25:46.494166125 +0000 UTC m=+1195.056492670"
Sep 09 03:25:46.578706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.578654    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:46.627187 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.627163    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:25:46.646141 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.646106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:46.685039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.684998    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-openshift-service-ca\") pod \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") "
Sep 09 03:25:46.685255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.685071    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crj4h\" (UniqueName: \"kubernetes.io/projected/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-kube-api-access-crj4h\") pod \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") "
Sep 09 03:25:46.685255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.685091    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-maas-api-tls\") pod \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") "
Sep 09 03:25:46.685255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.685134    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-metrics-certs\") pod \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\" (UID: \"67d5ed21-4e4a-4287-a01c-0a3bb93e3637\") "
Sep 09 03:25:46.685488 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.685462    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67d5ed21-4e4a-4287-a01c-0a3bb93e3637" (UID: "67d5ed21-4e4a-4287-a01c-0a3bb93e3637"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:46.687527 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.687494    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67d5ed21-4e4a-4287-a01c-0a3bb93e3637" (UID: "67d5ed21-4e4a-4287-a01c-0a3bb93e3637"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:46.687527 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.687513    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67d5ed21-4e4a-4287-a01c-0a3bb93e3637" (UID: "67d5ed21-4e4a-4287-a01c-0a3bb93e3637"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:46.687711 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.687616    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-kube-api-access-crj4h" (OuterVolumeSpecName: "kube-api-access-crj4h") pod "67d5ed21-4e4a-4287-a01c-0a3bb93e3637" (UID: "67d5ed21-4e4a-4287-a01c-0a3bb93e3637"). InnerVolumeSpecName "kube-api-access-crj4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:46.786912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.786876    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:46.786912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.786909    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crj4h\" (UniqueName: \"kubernetes.io/projected/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-kube-api-access-crj4h\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:46.786912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.786918    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:46.787158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.786927    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5ed21-4e4a-4287-a01c-0a3bb93e3637-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:25:46.942639 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:46.942523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:47.483205 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.483165    2562 generic.go:356] "Generic (PLEG): container finished" podID="67d5ed21-4e4a-4287-a01c-0a3bb93e3637" containerID="2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f" exitCode=0
Sep 09 03:25:47.483732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.483222    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"
Sep 09 03:25:47.483732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.483253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v" event={"ID":"67d5ed21-4e4a-4287-a01c-0a3bb93e3637","Type":"ContainerDied","Data":"2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f"}
Sep 09 03:25:47.483732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.483313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v" event={"ID":"67d5ed21-4e4a-4287-a01c-0a3bb93e3637","Type":"ContainerDied","Data":"fe9dedce34d446d8b021d2e8bc3ebd64203adc876fc2913ba2d85d300cbc09cc"}
Sep 09 03:25:47.483732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.483337    2562 scope.go:117] "RemoveContainer" containerID="2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f"
Sep 09 03:25:47.493622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.493598    2562 scope.go:117] "RemoveContainer" containerID="2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f"
Sep 09 03:25:47.493950 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:25:47.493925    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f\": container with ID starting with 2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f not found: ID does not exist" containerID="2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f"
Sep 09 03:25:47.494015 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.493959    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f"} err="failed to get container status \"2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f\": rpc error: code = NotFound desc = could not find container \"2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f\": container with ID starting with 2aa41ae19ede845c237acce444f42c18e93f27ec86b987196a422c77da27d82f not found: ID does not exist"
Sep 09 03:25:47.508117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.508086    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"]
Sep 09 03:25:47.512672 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.512643    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70784b-79d5859c7-nwf5v"]
Sep 09 03:25:47.938788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:47.938753    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d5ed21-4e4a-4287-a01c-0a3bb93e3637" path="/var/lib/kubelet/pods/67d5ed21-4e4a-4287-a01c-0a3bb93e3637/volumes"
Sep 09 03:25:52.489823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:52.489739    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:25:53.228386 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:53.228338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:53.388432 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:53.388382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:53.487748 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:53.487624    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:54.026137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.026098    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:54.294351 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.294310    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"]
Sep 09 03:25:54.294796 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.294781    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy"
Sep 09 03:25:54.294884 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.294798    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy"
Sep 09 03:25:54.294884 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.294806    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d5ed21-4e4a-4287-a01c-0a3bb93e3637" containerName="maas-api"
Sep 09 03:25:54.294884 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.294812    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d5ed21-4e4a-4287-a01c-0a3bb93e3637" containerName="maas-api"
Sep 09 03:25:54.295048 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.294894    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d5ed21-4e4a-4287-a01c-0a3bb93e3637" containerName="maas-api"
Sep 09 03:25:54.295048 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.294905    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c71dda-27bd-430a-8e0b-28a3f277f372" containerName="istio-proxy"
Sep 09 03:25:54.299496 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.299471    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.301793 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.301764    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1a85fa-openshift-default-dockercfg-g85jp\""
Sep 09 03:25:54.313167 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.312674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"]
Sep 09 03:25:54.325107 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.325052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:54.420905 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.420873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:54.474916 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.474879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-envoy\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.475109 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.474944    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-token\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.475109 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.475019    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ptx\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-kube-api-access-v8ptx\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.475109 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.475100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istiod-ca-cert\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.475240 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.475129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-socket\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.475240 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.475151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-podinfo\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.475330 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.475242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-data\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.475330 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.475308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-certs\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.475408 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.475343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-credential-socket\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.576740 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.576641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-envoy\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.576740 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.576692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-token\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.576740 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.576733    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ptx\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-kube-api-access-v8ptx\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.576976 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.576796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istiod-ca-cert\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.576976 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.576823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-socket\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-podinfo\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577184 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-socket\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-data\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-certs\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577303 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-credential-socket\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577641 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-data\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577654    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-certs\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-credential-socket\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.577804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.577688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istiod-ca-cert\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.579358 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.579331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-envoy\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.579951 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.579934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-podinfo\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.585833 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.585808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ptx\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-kube-api-access-v8ptx\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.587483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.587462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-token\") pod \"e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") " pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.614254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.614218    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:54.920268 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.920241    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"]
Sep 09 03:25:54.922810 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:25:54.922770    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81a990f_b8d2_43b2_8dbe_ee71d0f213b5.slice/crio-3d2fec05cccc1c993ceb2bc7b451d75cbb2b4c5ef0e7d94b23f23ac31d9a29ec WatchSource:0}: Error finding container 3d2fec05cccc1c993ceb2bc7b451d75cbb2b4c5ef0e7d94b23f23ac31d9a29ec: Status 404 returned error can't find the container with id 3d2fec05cccc1c993ceb2bc7b451d75cbb2b4c5ef0e7d94b23f23ac31d9a29ec
Sep 09 03:25:54.925201 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.925164    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:25:54.925313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.925244    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:25:54.925313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:54.925290    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:25:55.534372 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:55.534335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" event={"ID":"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5","Type":"ContainerStarted","Data":"afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650"}
Sep 09 03:25:55.534801 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:55.534394    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" event={"ID":"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5","Type":"ContainerStarted","Data":"3d2fec05cccc1c993ceb2bc7b451d75cbb2b4c5ef0e7d94b23f23ac31d9a29ec"}
Sep 09 03:25:55.614820 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:55.614782    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:56.620428 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:56.620396    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:56.684806 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:56.684741    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podStartSLOduration=2.684720946 podStartE2EDuration="2.684720946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:55.566197954 +0000 UTC m=+1204.128524500" watchObservedRunningTime="2026-09-09 03:25:56.684720946 +0000 UTC m=+1205.247047569"
Sep 09 03:25:57.499316 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:57.499275    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:57.549624 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:57.549591    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:57.551067 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:57.551041    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:25:57.597191 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:57.597159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:58.733801 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.733760    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"]
Sep 09 03:25:58.738528 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.738500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:58.741039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.740971    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1a85fa-serving-cert\""
Sep 09 03:25:58.761373 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.761334    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"]
Sep 09 03:25:58.790302 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.790262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:58.893635 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.893584    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:58.920035 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.919989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a8175d-04dc-45a2-84d6-224f1aa65f95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:58.920248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.920055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsk7\" (UniqueName: \"kubernetes.io/projected/65a8175d-04dc-45a2-84d6-224f1aa65f95-kube-api-access-zfsk7\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:58.920248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.920161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:58.920248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:58.920212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.021148 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.021050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.021148 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.021101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.021148 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.021136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a8175d-04dc-45a2-84d6-224f1aa65f95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.021477 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.021194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsk7\" (UniqueName: \"kubernetes.io/projected/65a8175d-04dc-45a2-84d6-224f1aa65f95-kube-api-access-zfsk7\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.022051 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.022025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a8175d-04dc-45a2-84d6-224f1aa65f95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.024050 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.024021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.024181 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.024021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.037330 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.037298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsk7\" (UniqueName: \"kubernetes.io/projected/65a8175d-04dc-45a2-84d6-224f1aa65f95-kube-api-access-zfsk7\") pod \"maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.065411 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.065372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:59.068311 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.068281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:25:59.350183 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.350155    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"]
Sep 09 03:25:59.352776 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:25:59.352729    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a8175d_04dc_45a2_84d6_224f1aa65f95.slice/crio-8c0e357c765b05aee2b1a45b96baf62cb5fe3e22215d74c0e165e2d87e0aea1c WatchSource:0}: Error finding container 8c0e357c765b05aee2b1a45b96baf62cb5fe3e22215d74c0e165e2d87e0aea1c: Status 404 returned error can't find the container with id 8c0e357c765b05aee2b1a45b96baf62cb5fe3e22215d74c0e165e2d87e0aea1c
Sep 09 03:25:59.471118 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.471072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:59.485788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.485754    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 03:25:59.485968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.485824    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 03:25:59.559989 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.559944    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:59.564948 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:25:59.564918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k" event={"ID":"65a8175d-04dc-45a2-84d6-224f1aa65f95","Type":"ContainerStarted","Data":"8c0e357c765b05aee2b1a45b96baf62cb5fe3e22215d74c0e165e2d87e0aea1c"}
Sep 09 03:26:00.572814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:00.572771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k" event={"ID":"65a8175d-04dc-45a2-84d6-224f1aa65f95","Type":"ContainerStarted","Data":"23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411"}
Sep 09 03:26:00.573314 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:00.572940    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:26:00.590706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:00.590653    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k" podStartSLOduration=2.122112879 podStartE2EDuration="2.590634893s" totalImagesPullingTime="468.522014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:00.589001758 +0000 UTC m=+1209.151328303" watchObservedRunningTime="2026-09-09 03:26:00.590634893 +0000 UTC m=+1209.152961438"
Sep 09 03:26:01.102711 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:01.102672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:01.581223 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:01.581183    2562 generic.go:356] "Generic (PLEG): container finished" podID="2e85f576-093f-4bb8-85ed-91636531b445" containerID="172e0124e24706435f342348817559733045cedd3cc6112247576d22b0a758bb" exitCode=0
Sep 09 03:26:01.581650 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:01.581256    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" event={"ID":"2e85f576-093f-4bb8-85ed-91636531b445","Type":"ContainerDied","Data":"172e0124e24706435f342348817559733045cedd3cc6112247576d22b0a758bb"}
Sep 09 03:26:01.581741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:01.581727    2562 scope.go:117] "RemoveContainer" containerID="172e0124e24706435f342348817559733045cedd3cc6112247576d22b0a758bb"
Sep 09 03:26:01.582789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:01.582762    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:26:01.582850 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:01.582836    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:26:01.582893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:01.582878    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:26:02.592758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:02.592723    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" event={"ID":"2e85f576-093f-4bb8-85ed-91636531b445","Type":"ContainerStarted","Data":"74bd58ce8b23ce98408c7941f2bf686fece5dc15de5b5daa3b4ab52bda303114"}
Sep 09 03:26:02.858759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:02.858649    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:26:02.859090 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:02.859060    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 03:26:02.859194 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:02.859125    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 03:26:03.858781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:03.858733    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 03:26:03.859276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:03.858810    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 03:26:04.859014 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:04.858971    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 03:26:04.860278 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:04.859659    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 03:26:05.559766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:05.559730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:05.858656 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:05.858508    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 03:26:05.858656 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:05.858634    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 03:26:06.595940 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:06.595896    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:26:06.858521 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:06.858404    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 03:26:06.858521 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:06.858501    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 03:26:07.858882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:07.858839    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 03:26:07.859398 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:07.858917    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 03:26:08.332772 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:08.332737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:08.613985 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:08.613899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:08.870712 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:08.870610    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:26:08.871840 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:08.871807    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:26:08.873092 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:08.873063    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:26:08.967091 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:08.967052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:09.216442 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:09.216334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:11.631141 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:11.631105    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:11.733173 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:11.733134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:12.732355 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.732320    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"]
Sep 09 03:26:12.734816 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.734794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.738443 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.738013    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4c8420-serving-cert\""
Sep 09 03:26:12.753029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.752922    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"]
Sep 09 03:26:12.778603 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.778569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:12.779584 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.779527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.779679 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.779597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.779679 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.779619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.779679 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.779651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66zz\" (UniqueName: \"kubernetes.io/projected/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-kube-api-access-p66zz\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.849808 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.849756    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": read tcp 10.133.0.2:43158->10.133.0.79:15021: read: connection reset by peer" start-of-body=
Sep 09 03:26:12.850012 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.849876    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": read tcp 10.133.0.2:43158->10.133.0.79:15021: read: connection reset by peer"
Sep 09 03:26:12.881078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.880921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.881078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.880987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66zz\" (UniqueName: \"kubernetes.io/projected/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-kube-api-access-p66zz\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.881324 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.881175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.881324 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.881195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.881324 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:26:12.881305    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-serving-cert: secret "maas-api-e2e-shared-b-w2-4c8420-serving-cert" not found
Sep 09 03:26:12.881442 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:26:12.881368    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls podName:83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc nodeName:}" failed. No retries permitted until 2026-09-09 03:26:13.381348906 +0000 UTC m=+1221.943675429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls") pod "maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr" (UID: "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc") : secret "maas-api-e2e-shared-b-w2-4c8420-serving-cert" not found
Sep 09 03:26:12.882210 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.882177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.883962 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.883937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.891630 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.891593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66zz\" (UniqueName: \"kubernetes.io/projected/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-kube-api-access-p66zz\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:12.917335 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:12.917302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:13.071358 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.071317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:13.386284 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.386182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:13.388946 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.388911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:13.656602 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.656459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:13.665160 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.665126    2562 generic.go:356] "Generic (PLEG): container finished" podID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerID="afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650" exitCode=0
Sep 09 03:26:13.665332 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.665213    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" event={"ID":"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5","Type":"ContainerDied","Data":"afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650"}
Sep 09 03:26:13.665744 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.665726    2562 scope.go:117] "RemoveContainer" containerID="afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650"
Sep 09 03:26:13.667079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.667044    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:26:13.667186 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.667130    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:26:13.667186 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.667177    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:26:13.684762 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.684726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:13.806938 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:13.806884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:14.000799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:14.000598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"]
Sep 09 03:26:14.677908 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:14.677811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" event={"ID":"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5","Type":"ContainerStarted","Data":"f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e"}
Sep 09 03:26:14.679686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:14.679654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr" event={"ID":"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc","Type":"ContainerStarted","Data":"402ee7a4ae80ec6419ea625e72896b1c7bb7ba16bbf0e69d859758d39640b7c1"}
Sep 09 03:26:14.679822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:14.679703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr" event={"ID":"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc","Type":"ContainerStarted","Data":"c44bf898265ebad893df10af08d05cc59d7b2b3f7eb63dcc01de2660328d0253"}
Sep 09 03:26:14.679822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:14.679734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:15.614973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:15.614931    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:26:15.615395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:15.615187    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 09 03:26:15.615395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:15.615240    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 09 03:26:16.615726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:16.615687    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 09 03:26:16.616122 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:16.615746    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 09 03:26:17.615943 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:17.615899    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 09 03:26:17.616359 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:17.615964    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 09 03:26:18.615026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:18.614982    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 09 03:26:18.615242 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:18.615047    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 09 03:26:19.615358 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:19.615324    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 09 03:26:19.615775 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:19.615394    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 09 03:26:20.083828 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:20.083795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:20.618593 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:20.618555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:26:20.618985 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:20.618919    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:26:20.619746 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:20.619724    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:26:20.657967 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:20.657908    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr" podStartSLOduration=8.361151051 podStartE2EDuration="8.657892218s" totalImagesPullingTime="296.741167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:14.753371486 +0000 UTC m=+1223.315698033" watchObservedRunningTime="2026-09-09 03:26:20.657892218 +0000 UTC m=+1229.220218775"
Sep 09 03:26:20.691353 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:20.691324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:26:22.503634 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:22.503600    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:22.907118 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:22.907080    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:23.454226 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:23.454188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:23.717901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:23.717798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:26.898429 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:26.898390    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:27.000666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:27.000627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:28.134440 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.134407    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"]
Sep 09 03:26:28.137961 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.137937    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.140364 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.140339    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-cd1da6-serving-cert\""
Sep 09 03:26:28.152069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.152041    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"]
Sep 09 03:26:28.217633 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.217588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:28.252350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.252307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8pl\" (UniqueName: \"kubernetes.io/projected/38a18f96-7589-4adc-86bc-fb6ae34041e4-kube-api-access-rm8pl\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.252529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.252363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.252529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.252497    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a18f96-7589-4adc-86bc-fb6ae34041e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.252676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.252528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.321960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.321928    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:28.353138 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.353096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8pl\" (UniqueName: \"kubernetes.io/projected/38a18f96-7589-4adc-86bc-fb6ae34041e4-kube-api-access-rm8pl\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.353138 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.353145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.353426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.353224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a18f96-7589-4adc-86bc-fb6ae34041e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.353426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.353355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.354006 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.353975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a18f96-7589-4adc-86bc-fb6ae34041e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.355865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.355841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.355998 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.355934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.363593 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.363564    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8pl\" (UniqueName: \"kubernetes.io/projected/38a18f96-7589-4adc-86bc-fb6ae34041e4-kube-api-access-rm8pl\") pod \"maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.452310 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.452228    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:28.741310 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.741287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"]
Sep 09 03:26:28.742102 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:26:28.742059    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a18f96_7589_4adc_86bc_fb6ae34041e4.slice/crio-ec89f681dc392be5910f78a3cc56a013123d3a7d861d8b83260683b3128ccd76 WatchSource:0}: Error finding container ec89f681dc392be5910f78a3cc56a013123d3a7d861d8b83260683b3128ccd76: Status 404 returned error can't find the container with id ec89f681dc392be5910f78a3cc56a013123d3a7d861d8b83260683b3128ccd76
Sep 09 03:26:28.770828 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.770790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz" event={"ID":"38a18f96-7589-4adc-86bc-fb6ae34041e4","Type":"ContainerStarted","Data":"ec89f681dc392be5910f78a3cc56a013123d3a7d861d8b83260683b3128ccd76"}
Sep 09 03:26:28.832931 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.832896    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:28.933073 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:28.933044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:29.779639 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:29.779603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz" event={"ID":"38a18f96-7589-4adc-86bc-fb6ae34041e4","Type":"ContainerStarted","Data":"baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc"}
Sep 09 03:26:29.780052 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:29.779769    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:29.799427 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:29.799378    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz" podStartSLOduration=1.418152573 podStartE2EDuration="1.799361145s" totalImagesPullingTime="381.208572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:29.796885627 +0000 UTC m=+1238.359212174" watchObservedRunningTime="2026-09-09 03:26:29.799361145 +0000 UTC m=+1238.361687690"
Sep 09 03:26:34.310248 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:34.310213    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:35.791079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:35.791051    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:26:36.563017 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:36.562982    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:49.352987 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:49.352944    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:50.850238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:50.850195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:51.171962 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:51.171862    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:51.375197 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:51.375165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:51.732611 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:51.732572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:52.106068 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:52.106025    2562 scope.go:117] "RemoveContainer" containerID="6bea4595886be94868cd378b8d93ad98fa39dd00262a28cd81c7e10517a7819d"
Sep 09 03:26:53.993078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:53.993042    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:54.094416 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:54.094364    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:55.197654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.197617    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"]
Sep 09 03:26:55.203473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.203450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.207322 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.207295    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a3a90070-serving-cert\""
Sep 09 03:26:55.222134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.222100    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"]
Sep 09 03:26:55.288123 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.288084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:55.332344 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.332312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-metrics-certs\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.332344 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.332352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwz4z\" (UniqueName: \"kubernetes.io/projected/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-kube-api-access-mwz4z\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.332599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.332446    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-openshift-service-ca\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.332599 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.332508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-maas-api-tls\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.408259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.408216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:55.433860 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.433818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-openshift-service-ca\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.434076 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.433900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-maas-api-tls\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.434076 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.433966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-metrics-certs\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.434076 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.433990    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwz4z\" (UniqueName: \"kubernetes.io/projected/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-kube-api-access-mwz4z\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.434628 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.434605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-openshift-service-ca\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.437198 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.437175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-maas-api-tls\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.437487 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.437463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-metrics-certs\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.446333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.446307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwz4z\" (UniqueName: \"kubernetes.io/projected/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-kube-api-access-mwz4z\") pod \"maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.516896 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.516797    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:55.571861 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.571818    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:55.820938 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.820909    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"]
Sep 09 03:26:55.823138 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:26:55.823113    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6292e9b5_b8bc_43ef_9ea0_d26f54b02cfb.slice/crio-fa9a9d2d655130d4e3113f0b4180e5fe5ab3b398ccb01073aaa9b73a6cb0cbf1 WatchSource:0}: Error finding container fa9a9d2d655130d4e3113f0b4180e5fe5ab3b398ccb01073aaa9b73a6cb0cbf1: Status 404 returned error can't find the container with id fa9a9d2d655130d4e3113f0b4180e5fe5ab3b398ccb01073aaa9b73a6cb0cbf1
Sep 09 03:26:55.953454 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:55.953406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg" event={"ID":"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb","Type":"ContainerStarted","Data":"fa9a9d2d655130d4e3113f0b4180e5fe5ab3b398ccb01073aaa9b73a6cb0cbf1"}
Sep 09 03:26:56.063420 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:56.063387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:56.194313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:56.194221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:56.962210 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:56.962171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg" event={"ID":"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb","Type":"ContainerStarted","Data":"a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195"}
Sep 09 03:26:56.962657 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:56.962272    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:26:56.986342 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:26:56.986291    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg" podStartSLOduration=1.5111025150000001 podStartE2EDuration="1.986274451s" totalImagesPullingTime="475.171936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:56.986057011 +0000 UTC m=+1265.548383555" watchObservedRunningTime="2026-09-09 03:26:56.986274451 +0000 UTC m=+1265.548600996"
Sep 09 03:27:02.974225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:02.974188    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:27:05.663368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:05.663324    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:27:19.000700 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:19.000659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:27:36.347487 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.347441    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"]
Sep 09 03:27:36.351432 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.351413    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.354840 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.354814    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a3a90070\"/\"default-dockercfg-q7twn\""
Sep 09 03:27:36.354994 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.354877    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a3a90070\"/\"openshift-service-ca.crt\""
Sep 09 03:27:36.354994 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.354813    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a3a90070\"/\"e2e-lifecycle-model-a3a90070-kserve-self-signed-certs\""
Sep 09 03:27:36.354994 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.354826    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a3a90070\"/\"kube-root-ca.crt\""
Sep 09 03:27:36.364032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.363999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqxhn\" (UniqueName: \"kubernetes.io/projected/b277b93e-9650-499a-ab3d-1d821cb31bfd-kube-api-access-rqxhn\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.364216 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.364192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-tmp-dir\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.364290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.364254    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-home\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.364349 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.364305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b277b93e-9650-499a-ab3d-1d821cb31bfd-tls-certs\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.364411 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.364361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-model-cache\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.364469 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.364405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-dshm\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.376424 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.376386    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"]
Sep 09 03:27:36.396302 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.396270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:27:36.465642 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.465553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqxhn\" (UniqueName: \"kubernetes.io/projected/b277b93e-9650-499a-ab3d-1d821cb31bfd-kube-api-access-rqxhn\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.465642 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.465593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-tmp-dir\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.465642 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.465620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-home\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.465642 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.465654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b277b93e-9650-499a-ab3d-1d821cb31bfd-tls-certs\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.466045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.465682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-model-cache\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.466045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.465710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-dshm\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.466169 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.466060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-tmp-dir\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.466169 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.466073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-model-cache\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.466169 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.466131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-home\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.468321 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.468291    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-dshm\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.468445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.468403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b277b93e-9650-499a-ab3d-1d821cb31bfd-tls-certs\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.475771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.475737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqxhn\" (UniqueName: \"kubernetes.io/projected/b277b93e-9650-499a-ab3d-1d821cb31bfd-kube-api-access-rqxhn\") pod \"e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") " pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.666949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.666856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:36.949731 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:27:36.949690    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb277b93e_9650_499a_ab3d_1d821cb31bfd.slice/crio-8d081da0167fac53cfd5f12d0547c996a7d18571f79d781dcc5cd2bae0f20b68 WatchSource:0}: Error finding container 8d081da0167fac53cfd5f12d0547c996a7d18571f79d781dcc5cd2bae0f20b68: Status 404 returned error can't find the container with id 8d081da0167fac53cfd5f12d0547c996a7d18571f79d781dcc5cd2bae0f20b68
Sep 09 03:27:36.952917 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:36.952887    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"]
Sep 09 03:27:37.215350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:37.215242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9" event={"ID":"b277b93e-9650-499a-ab3d-1d821cb31bfd","Type":"ContainerStarted","Data":"c3228a3ab3e1af9212ab447768f74e408a516c445a1813e1c7109d3a90bf84cf"}
Sep 09 03:27:37.215350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:37.215288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9" event={"ID":"b277b93e-9650-499a-ab3d-1d821cb31bfd","Type":"ContainerStarted","Data":"8d081da0167fac53cfd5f12d0547c996a7d18571f79d781dcc5cd2bae0f20b68"}
Sep 09 03:27:37.236361 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:37.236300    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9" podStartSLOduration=1.236277747 podStartE2EDuration="1.236277747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:27:37.235267933 +0000 UTC m=+1305.797594480" watchObservedRunningTime="2026-09-09 03:27:37.236277747 +0000 UTC m=+1305.798604291"
Sep 09 03:27:45.766023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:45.765984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:27:45.867300 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:45.867263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:27:46.667779 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:46.667738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:46.667980 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:46.667798    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:46.680680 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:46.680649    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:47.306889 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:47.306861    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:27:50.082209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:50.082166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:27:56.115659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:27:56.115614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:24.784441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:24.784343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:40.145878 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.145843    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"]
Sep 09 03:28:40.146369 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.146065    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg" podUID="6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" containerName="maas-api" containerID="cri-o://a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195" gracePeriod=30
Sep 09 03:28:40.183110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.183081    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:40.280743 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.280680    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:40.362623 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.362592    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:40.409772 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.409711    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:28:40.440780 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.440750    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-maas-api-tls\") pod \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") "
Sep 09 03:28:40.440970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.440794    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwz4z\" (UniqueName: \"kubernetes.io/projected/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-kube-api-access-mwz4z\") pod \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") "
Sep 09 03:28:40.440970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.440813    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-openshift-service-ca\") pod \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") "
Sep 09 03:28:40.440970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.440846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-metrics-certs\") pod \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\" (UID: \"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb\") "
Sep 09 03:28:40.441235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.441201    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" (UID: "6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:28:40.443629 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.443599    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" (UID: "6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:28:40.443790 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.443764    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-kube-api-access-mwz4z" (OuterVolumeSpecName: "kube-api-access-mwz4z") pod "6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" (UID: "6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb"). InnerVolumeSpecName "kube-api-access-mwz4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:28:40.443790 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.443779    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" (UID: "6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:28:40.468397 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.468362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:40.536526 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.536486    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:40.541915 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.541889    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:28:40.542035 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.541921    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwz4z\" (UniqueName: \"kubernetes.io/projected/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-kube-api-access-mwz4z\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:28:40.542035 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.541935    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:28:40.542035 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.541949    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:28:40.633500 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.633462    2562 generic.go:356] "Generic (PLEG): container finished" podID="6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" containerID="a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195" exitCode=0
Sep 09 03:28:40.633500 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.633493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg" event={"ID":"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb","Type":"ContainerDied","Data":"a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195"}
Sep 09 03:28:40.633789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.633508    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"
Sep 09 03:28:40.633789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.633582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg" event={"ID":"6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb","Type":"ContainerDied","Data":"fa9a9d2d655130d4e3113f0b4180e5fe5ab3b398ccb01073aaa9b73a6cb0cbf1"}
Sep 09 03:28:40.633789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.633617    2562 scope.go:117] "RemoveContainer" containerID="a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195"
Sep 09 03:28:40.644532 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.644509    2562 scope.go:117] "RemoveContainer" containerID="a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195"
Sep 09 03:28:40.644912 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:28:40.644890    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195\": container with ID starting with a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195 not found: ID does not exist" containerID="a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195"
Sep 09 03:28:40.645002 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.644921    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195"} err="failed to get container status \"a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195\": rpc error: code = NotFound desc = could not find container \"a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195\": container with ID starting with a6e2a936a29b69be1d6212913f08abf1a7e0a16e573bd0125159d3fb5382c195 not found: ID does not exist"
Sep 09 03:28:40.662445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.662358    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"]
Sep 09 03:28:40.667343 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:40.667315    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3a90070-6cdf4896d5-kd9jg"]
Sep 09 03:28:41.939681 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:41.939646    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" path="/var/lib/kubelet/pods/6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb/volumes"
Sep 09 03:28:43.646570 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:43.646505    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:48.245688 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:48.245648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:48.373184 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:48.373150    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:48.623507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:48.623467    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"]
Sep 09 03:28:48.720255 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:28:48.720211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:03.644928 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.644861    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9" podUID="b277b93e-9650-499a-ab3d-1d821cb31bfd" containerName="main" containerID="cri-o://c3228a3ab3e1af9212ab447768f74e408a516c445a1813e1c7109d3a90bf84cf" gracePeriod=45
Sep 09 03:29:03.783790 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.783756    2562 generic.go:356] "Generic (PLEG): container finished" podID="b277b93e-9650-499a-ab3d-1d821cb31bfd" containerID="c3228a3ab3e1af9212ab447768f74e408a516c445a1813e1c7109d3a90bf84cf" exitCode=0
Sep 09 03:29:03.784021 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.783818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9" event={"ID":"b277b93e-9650-499a-ab3d-1d821cb31bfd","Type":"ContainerDied","Data":"c3228a3ab3e1af9212ab447768f74e408a516c445a1813e1c7109d3a90bf84cf"}
Sep 09 03:29:03.909606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.909519    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:29:03.983358 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983315    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-model-cache\") pod \"b277b93e-9650-499a-ab3d-1d821cb31bfd\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") "
Sep 09 03:29:03.983591 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983387    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-tmp-dir\") pod \"b277b93e-9650-499a-ab3d-1d821cb31bfd\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") "
Sep 09 03:29:03.983591 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983419    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqxhn\" (UniqueName: \"kubernetes.io/projected/b277b93e-9650-499a-ab3d-1d821cb31bfd-kube-api-access-rqxhn\") pod \"b277b93e-9650-499a-ab3d-1d821cb31bfd\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") "
Sep 09 03:29:03.983591 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983462    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-home\") pod \"b277b93e-9650-499a-ab3d-1d821cb31bfd\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") "
Sep 09 03:29:03.983591 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983494    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-dshm\") pod \"b277b93e-9650-499a-ab3d-1d821cb31bfd\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") "
Sep 09 03:29:03.983832 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983653    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b277b93e-9650-499a-ab3d-1d821cb31bfd-tls-certs\") pod \"b277b93e-9650-499a-ab3d-1d821cb31bfd\" (UID: \"b277b93e-9650-499a-ab3d-1d821cb31bfd\") "
Sep 09 03:29:03.983832 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b277b93e-9650-499a-ab3d-1d821cb31bfd" (UID: "b277b93e-9650-499a-ab3d-1d821cb31bfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:03.983832 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983759    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b277b93e-9650-499a-ab3d-1d821cb31bfd" (UID: "b277b93e-9650-499a-ab3d-1d821cb31bfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:03.983832 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.983780    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-home" (OuterVolumeSpecName: "home") pod "b277b93e-9650-499a-ab3d-1d821cb31bfd" (UID: "b277b93e-9650-499a-ab3d-1d821cb31bfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:03.984036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.984011    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:29:03.984036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.984036    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:29:03.984204 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.984048    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:29:03.986118 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.986092    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b277b93e-9650-499a-ab3d-1d821cb31bfd-kube-api-access-rqxhn" (OuterVolumeSpecName: "kube-api-access-rqxhn") pod "b277b93e-9650-499a-ab3d-1d821cb31bfd" (UID: "b277b93e-9650-499a-ab3d-1d821cb31bfd"). InnerVolumeSpecName "kube-api-access-rqxhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:29:03.986483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.986454    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-dshm" (OuterVolumeSpecName: "dshm") pod "b277b93e-9650-499a-ab3d-1d821cb31bfd" (UID: "b277b93e-9650-499a-ab3d-1d821cb31bfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:03.986483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:03.986454    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b277b93e-9650-499a-ab3d-1d821cb31bfd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b277b93e-9650-499a-ab3d-1d821cb31bfd" (UID: "b277b93e-9650-499a-ab3d-1d821cb31bfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:29:04.085317 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:04.085268    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b277b93e-9650-499a-ab3d-1d821cb31bfd-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:29:04.085317 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:04.085308    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqxhn\" (UniqueName: \"kubernetes.io/projected/b277b93e-9650-499a-ab3d-1d821cb31bfd-kube-api-access-rqxhn\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:29:04.085317 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:04.085318    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b277b93e-9650-499a-ab3d-1d821cb31bfd-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:29:04.791411 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:04.791380    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"
Sep 09 03:29:04.791883 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:04.791374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9" event={"ID":"b277b93e-9650-499a-ab3d-1d821cb31bfd","Type":"ContainerDied","Data":"8d081da0167fac53cfd5f12d0547c996a7d18571f79d781dcc5cd2bae0f20b68"}
Sep 09 03:29:04.791883 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:04.791512    2562 scope.go:117] "RemoveContainer" containerID="c3228a3ab3e1af9212ab447768f74e408a516c445a1813e1c7109d3a90bf84cf"
Sep 09 03:29:04.816027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:04.815990    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"]
Sep 09 03:29:04.823787 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:04.823750    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a3a90070/e2e-lifecycle-model-a3a90070-kserve-845db5c8c5-ntxp9"]
Sep 09 03:29:05.938952 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:05.938914    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b277b93e-9650-499a-ab3d-1d821cb31bfd" path="/var/lib/kubelet/pods/b277b93e-9650-499a-ab3d-1d821cb31bfd/volumes"
Sep 09 03:29:26.168146 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:26.168111    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:26.270171 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:26.270130    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:31.727926 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:31.727893    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:31.843624 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:31.843581    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:32.065441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:32.065406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:33.397970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:33.397928    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:33.585812 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:33.585779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:34.154063 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:34.154018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:34.250179 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:34.250129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:34.350736 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:34.350699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:35.577136 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:35.577091    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:41.410880 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:41.410851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:41.485822 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:41.485780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:42.146998 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:42.146949    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:47.970418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:47.970379    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:48.086382 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:48.086346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:48.192161 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:48.192125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:58.705919 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:58.705874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:58.895854 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:29:58.895817    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:19.814817 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:19.814781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:20.921494 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.921450    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7604-de255d"]
Sep 09 03:30:20.921964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.921947    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" containerName="maas-api"
Sep 09 03:30:20.921964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.921965    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" containerName="maas-api"
Sep 09 03:30:20.922045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.921978    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b277b93e-9650-499a-ab3d-1d821cb31bfd" containerName="main"
Sep 09 03:30:20.922045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.921983    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b277b93e-9650-499a-ab3d-1d821cb31bfd" containerName="main"
Sep 09 03:30:20.922111 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.922072    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6292e9b5-b8bc-43ef-9ea0-d26f54b02cfb" containerName="maas-api"
Sep 09 03:30:20.922111 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.922083    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b277b93e-9650-499a-ab3d-1d821cb31bfd" containerName="main"
Sep 09 03:30:20.925344 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.925298    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-de255d"
Sep 09 03:30:20.927746 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.927716    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-49nx9\""
Sep 09 03:30:20.929618 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:20.929585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7604-de255d"]
Sep 09 03:30:21.072960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:21.072917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5f8\" (UniqueName: \"kubernetes.io/projected/136dc998-91d1-4412-b5a3-b53538724500-kube-api-access-qd5f8\") pod \"mt-curl-7604-de255d\" (UID: \"136dc998-91d1-4412-b5a3-b53538724500\") " pod="opendatahub/mt-curl-7604-de255d"
Sep 09 03:30:21.174119 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:21.174010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5f8\" (UniqueName: \"kubernetes.io/projected/136dc998-91d1-4412-b5a3-b53538724500-kube-api-access-qd5f8\") pod \"mt-curl-7604-de255d\" (UID: \"136dc998-91d1-4412-b5a3-b53538724500\") " pod="opendatahub/mt-curl-7604-de255d"
Sep 09 03:30:21.183176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:21.183136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5f8\" (UniqueName: \"kubernetes.io/projected/136dc998-91d1-4412-b5a3-b53538724500-kube-api-access-qd5f8\") pod \"mt-curl-7604-de255d\" (UID: \"136dc998-91d1-4412-b5a3-b53538724500\") " pod="opendatahub/mt-curl-7604-de255d"
Sep 09 03:30:21.237763 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:21.237729    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-de255d"
Sep 09 03:30:21.515556 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:21.515500    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7604-de255d"]
Sep 09 03:30:21.516803 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:30:21.516774    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod136dc998_91d1_4412_b5a3_b53538724500.slice/crio-d5988f269b9a0362deeedb40f8ddc29813f41fb51e2795206b1544760e0218e4 WatchSource:0}: Error finding container d5988f269b9a0362deeedb40f8ddc29813f41fb51e2795206b1544760e0218e4: Status 404 returned error can't find the container with id d5988f269b9a0362deeedb40f8ddc29813f41fb51e2795206b1544760e0218e4
Sep 09 03:30:21.518137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:21.518119    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:30:22.271505 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:22.271465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7604-de255d" event={"ID":"136dc998-91d1-4412-b5a3-b53538724500","Type":"ContainerStarted","Data":"ccb5fcc9e9a98b354bcf33adf868b2f8f95d5403928028031279370c4b916eea"}
Sep 09 03:30:22.271943 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:22.271519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7604-de255d" event={"ID":"136dc998-91d1-4412-b5a3-b53538724500","Type":"ContainerStarted","Data":"d5988f269b9a0362deeedb40f8ddc29813f41fb51e2795206b1544760e0218e4"}
Sep 09 03:30:22.287878 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:22.287810    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/mt-curl-7604-de255d" podStartSLOduration=1.614855557 podStartE2EDuration="2.287789612s" totalImagesPullingTime="672.934055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:22.28628126 +0000 UTC m=+1470.848607805" watchObservedRunningTime="2026-09-09 03:30:22.287789612 +0000 UTC m=+1470.850116174"
Sep 09 03:30:22.385852 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:22.385820    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7604-de255d_136dc998-91d1-4412-b5a3-b53538724500/mt-curl-7604-de255d/0.log"
Sep 09 03:30:23.279366 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:23.279332    2562 generic.go:356] "Generic (PLEG): container finished" podID="136dc998-91d1-4412-b5a3-b53538724500" containerID="ccb5fcc9e9a98b354bcf33adf868b2f8f95d5403928028031279370c4b916eea" exitCode=0
Sep 09 03:30:23.279813 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:23.279420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7604-de255d" event={"ID":"136dc998-91d1-4412-b5a3-b53538724500","Type":"ContainerDied","Data":"ccb5fcc9e9a98b354bcf33adf868b2f8f95d5403928028031279370c4b916eea"}
Sep 09 03:30:24.426551 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:24.426508    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-de255d"
Sep 09 03:30:24.461060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:24.461024    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7604-de255d"]
Sep 09 03:30:24.464600 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:24.464571    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7604-de255d"]
Sep 09 03:30:24.608039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:24.607896    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd5f8\" (UniqueName: \"kubernetes.io/projected/136dc998-91d1-4412-b5a3-b53538724500-kube-api-access-qd5f8\") pod \"136dc998-91d1-4412-b5a3-b53538724500\" (UID: \"136dc998-91d1-4412-b5a3-b53538724500\") "
Sep 09 03:30:24.610456 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:24.610426    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136dc998-91d1-4412-b5a3-b53538724500-kube-api-access-qd5f8" (OuterVolumeSpecName: "kube-api-access-qd5f8") pod "136dc998-91d1-4412-b5a3-b53538724500" (UID: "136dc998-91d1-4412-b5a3-b53538724500"). InnerVolumeSpecName "kube-api-access-qd5f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:24.709373 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:24.709332    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd5f8\" (UniqueName: \"kubernetes.io/projected/136dc998-91d1-4412-b5a3-b53538724500-kube-api-access-qd5f8\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:30:25.292414 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:25.292383    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-de255d"
Sep 09 03:30:25.292619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:25.292437    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5988f269b9a0362deeedb40f8ddc29813f41fb51e2795206b1544760e0218e4"
Sep 09 03:30:25.302917 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:30:25.302885    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7604-de255d\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="136dc998-91d1-4412-b5a3-b53538724500" pod="opendatahub/mt-curl-7604-de255d"
Sep 09 03:30:25.825052 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:25.825014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:25.938350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:25.938311    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136dc998-91d1-4412-b5a3-b53538724500" path="/var/lib/kubelet/pods/136dc998-91d1-4412-b5a3-b53538724500/volumes"
Sep 09 03:30:26.096373 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.096268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:26.154849 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.154804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:26.439810 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.439713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:26.725426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.725336    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"]
Sep 09 03:30:26.726005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.725985    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136dc998-91d1-4412-b5a3-b53538724500" containerName="mt-curl-7604-de255d"
Sep 09 03:30:26.726079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.726007    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="136dc998-91d1-4412-b5a3-b53538724500" containerName="mt-curl-7604-de255d"
Sep 09 03:30:26.726136 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.726099    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="136dc998-91d1-4412-b5a3-b53538724500" containerName="mt-curl-7604-de255d"
Sep 09 03:30:26.729489 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.729471    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.732772 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.732755    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ec5029-openshift-default-dockercfg-585f2\""
Sep 09 03:30:26.744618 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.744590    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:26.746219 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.746192    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"]
Sep 09 03:30:26.832264 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-envoy\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.832807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istiod-ca-cert\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.832807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832342    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-podinfo\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.832807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-token\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.832807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-credential-socket\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.832807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-data\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.832807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832557    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4496\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-kube-api-access-n4496\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.832807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-certs\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.832807 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.832697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-socket\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933492 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-credential-socket\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-data\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4496\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-kube-api-access-n4496\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-certs\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933732 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-socket\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933984 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-envoy\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933984 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istiod-ca-cert\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933984 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-podinfo\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933984 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933855    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-token\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.933984 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-credential-socket\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.934228 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.933993    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-data\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.934228 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.934124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-certs\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.934429 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.934398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-socket\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.934591 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.934505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istiod-ca-cert\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.936554 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.936524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-envoy\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.936686 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.936666    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-podinfo\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.941945 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.941919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-token\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:26.942170 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:26.942147    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4496\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-kube-api-access-n4496\") pod \"e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") " pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:27.040135 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:27.040095    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:27.130483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:27.130370    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:27.223768 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:27.223717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:27.325244 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:27.325217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"]
Sep 09 03:30:27.327177 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:30:27.327150    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fa7df2_d9c2_4bcb_897b_985c29cfb716.slice/crio-2175aec34ddd9dc060222b6268c6912b36dbbe881f127368836d9977083c1504 WatchSource:0}: Error finding container 2175aec34ddd9dc060222b6268c6912b36dbbe881f127368836d9977083c1504: Status 404 returned error can't find the container with id 2175aec34ddd9dc060222b6268c6912b36dbbe881f127368836d9977083c1504
Sep 09 03:30:27.329558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:27.329508    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:30:27.329666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:27.329610    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:30:27.329666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:27.329654    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:30:28.313027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:28.312989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" event={"ID":"70fa7df2-d9c2-4bcb-897b-985c29cfb716","Type":"ContainerStarted","Data":"6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716"}
Sep 09 03:30:28.313444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:28.313046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" event={"ID":"70fa7df2-d9c2-4bcb-897b-985c29cfb716","Type":"ContainerStarted","Data":"2175aec34ddd9dc060222b6268c6912b36dbbe881f127368836d9977083c1504"}
Sep 09 03:30:28.337478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:28.337419    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" podStartSLOduration=2.337398084 podStartE2EDuration="2.337398084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:28.334713601 +0000 UTC m=+1476.897040170" watchObservedRunningTime="2026-09-09 03:30:28.337398084 +0000 UTC m=+1476.899724629"
Sep 09 03:30:29.040744 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:29.040701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:29.046365 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:29.046333    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:29.319742 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:29.319647    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:29.320806 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:29.320786    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:30.259323 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:30.259277    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:30.339616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:30.339573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:31.350615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.350574    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"]
Sep 09 03:30:31.355241 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.355213    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.357512 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.357483    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ec5029-serving-cert\""
Sep 09 03:30:31.365131 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.365091    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"]
Sep 09 03:30:31.376212 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.376175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-metrics-certs\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.376406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.376238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3107598a-5ae3-452a-8fb6-4764aa088008-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.376406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.376284    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.376582 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.376558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9ph\" (UniqueName: \"kubernetes.io/projected/3107598a-5ae3-452a-8fb6-4764aa088008-kube-api-access-tg9ph\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.385610 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.385576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:31.477415 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.477375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-metrics-certs\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.477670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.477424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3107598a-5ae3-452a-8fb6-4764aa088008-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.477670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.477467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.477670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.477614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9ph\" (UniqueName: \"kubernetes.io/projected/3107598a-5ae3-452a-8fb6-4764aa088008-kube-api-access-tg9ph\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.477670 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:30:31.477656    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-serving-cert: secret "maas-api-e2e-api-a-ec5029-serving-cert" not found
Sep 09 03:30:31.477924 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:30:31.477748    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls podName:3107598a-5ae3-452a-8fb6-4764aa088008 nodeName:}" failed. No retries permitted until 2026-09-09 03:30:31.97772508 +0000 UTC m=+1480.540051604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls") pod "maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf" (UID: "3107598a-5ae3-452a-8fb6-4764aa088008") : secret "maas-api-e2e-api-a-ec5029-serving-cert" not found
Sep 09 03:30:31.478239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.478215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3107598a-5ae3-452a-8fb6-4764aa088008-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.480439 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.480414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-metrics-certs\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.491117 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.491077    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:31.498045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.498018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9ph\" (UniqueName: \"kubernetes.io/projected/3107598a-5ae3-452a-8fb6-4764aa088008-kube-api-access-tg9ph\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.692375 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.692266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:31.883235 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.883201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:31.983619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.983499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.986656 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.986596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls\") pod \"maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:31.988023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:31.987991    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:32.269170 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:32.269082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:32.557466 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:32.557431    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"]
Sep 09 03:30:32.559633 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:30:32.559607    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3107598a_5ae3_452a_8fb6_4764aa088008.slice/crio-f25d7804750478a71eab1263b4c34a4aab45263d5f6747ec816ec4b23bdb212e WatchSource:0}: Error finding container f25d7804750478a71eab1263b4c34a4aab45263d5f6747ec816ec4b23bdb212e: Status 404 returned error can't find the container with id f25d7804750478a71eab1263b4c34a4aab45263d5f6747ec816ec4b23bdb212e
Sep 09 03:30:33.366211 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:33.366162    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf" event={"ID":"3107598a-5ae3-452a-8fb6-4764aa088008","Type":"ContainerStarted","Data":"03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5"}
Sep 09 03:30:33.366422 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:33.366236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf" event={"ID":"3107598a-5ae3-452a-8fb6-4764aa088008","Type":"ContainerStarted","Data":"f25d7804750478a71eab1263b4c34a4aab45263d5f6747ec816ec4b23bdb212e"}
Sep 09 03:30:33.366422 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:33.366267    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:33.387251 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:33.387184    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf" podStartSLOduration=1.7303048749999999 podStartE2EDuration="2.387164977s" totalImagesPullingTime="656.860102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:33.384605799 +0000 UTC m=+1481.946932358" watchObservedRunningTime="2026-09-09 03:30:33.387164977 +0000 UTC m=+1481.949491561"
Sep 09 03:30:39.377774 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:39.377741    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:30:39.747758 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:39.747664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:51.494257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:51.494220    2562 generic.go:356] "Generic (PLEG): container finished" podID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerID="6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716" exitCode=0
Sep 09 03:30:51.494859 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:51.494299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" event={"ID":"70fa7df2-d9c2-4bcb-897b-985c29cfb716","Type":"ContainerDied","Data":"6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716"}
Sep 09 03:30:51.494943 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:51.494869    2562 scope.go:117] "RemoveContainer" containerID="6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716"
Sep 09 03:30:51.496079 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:51.496046    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:30:51.496182 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:51.496120    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:30:51.496182 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:51.496153    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:30:52.503033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:52.502994    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" event={"ID":"70fa7df2-d9c2-4bcb-897b-985c29cfb716","Type":"ContainerStarted","Data":"495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e"}
Sep 09 03:30:53.040467 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:53.040427    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:53.040669 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:53.040649    2562 patch_prober.go:28] interesting pod/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 09 03:30:53.040738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:53.040716    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 09 03:30:54.041427 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:54.041393    2562 patch_prober.go:28] interesting pod/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 09 03:30:54.042218 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:54.041465    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 09 03:30:55.041340 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:55.041301    2562 patch_prober.go:28] interesting pod/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 09 03:30:55.041581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:55.041365    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 09 03:30:55.225087 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:55.225047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:55.710947 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:55.710912    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:56.040598 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:56.040535    2562 patch_prober.go:28] interesting pod/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 09 03:30:56.040811 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:56.040628    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 09 03:30:56.041514 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:56.041492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:56.551397 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:56.551361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:57.041425 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:57.041395    2562 patch_prober.go:28] interesting pod/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 09 03:30:57.041669 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:57.041458    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 09 03:30:58.046150 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:58.046111    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:58.046571 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:58.046527    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:58.047463 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:58.047442    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:30:59.481259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:59.481223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:59.563923 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:30:59.563885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:00.562617 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.562568    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"]
Sep 09 03:31:00.568137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.568110    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.570970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.570943    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-811c96-serving-cert\""
Sep 09 03:31:00.583503 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.583474    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"]
Sep 09 03:31:00.602735 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.602697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:00.707225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.707187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:00.711385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.711362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-metrics-certs\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.711525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.711481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1557cc59-1769-4de0-ba8a-e94c3394b3e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.711525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.711519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-maas-api-tls\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.711649 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.711579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxn7v\" (UniqueName: \"kubernetes.io/projected/1557cc59-1769-4de0-ba8a-e94c3394b3e3-kube-api-access-fxn7v\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.801112 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.801072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:00.812317 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.812287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-metrics-certs\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.812496 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.812375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1557cc59-1769-4de0-ba8a-e94c3394b3e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.812496 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.812406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-maas-api-tls\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.812496 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.812429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxn7v\" (UniqueName: \"kubernetes.io/projected/1557cc59-1769-4de0-ba8a-e94c3394b3e3-kube-api-access-fxn7v\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.813213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.813151    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1557cc59-1769-4de0-ba8a-e94c3394b3e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.815157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.815134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-maas-api-tls\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.815157 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.815149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-metrics-certs\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.820314 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.820290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxn7v\" (UniqueName: \"kubernetes.io/projected/1557cc59-1769-4de0-ba8a-e94c3394b3e3-kube-api-access-fxn7v\") pod \"maas-api-e2e-api-b-811c96-6958dd7cf6-94586\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:00.884741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:00.884697    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:01.151352 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:01.151260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:01.170520 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:01.170471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"]
Sep 09 03:31:01.171982 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:31:01.171954    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1557cc59_1769_4de0_ba8a_e94c3394b3e3.slice/crio-6a9a2bde671b8742b8b256e46cf6c315653066e7b468d199cba56cf1671dcc87 WatchSource:0}: Error finding container 6a9a2bde671b8742b8b256e46cf6c315653066e7b468d199cba56cf1671dcc87: Status 404 returned error can't find the container with id 6a9a2bde671b8742b8b256e46cf6c315653066e7b468d199cba56cf1671dcc87
Sep 09 03:31:01.242233 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:01.242196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:01.563625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:01.563580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586" event={"ID":"1557cc59-1769-4de0-ba8a-e94c3394b3e3","Type":"ContainerStarted","Data":"6a9a2bde671b8742b8b256e46cf6c315653066e7b468d199cba56cf1671dcc87"}
Sep 09 03:31:02.571569 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:02.571516    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586" event={"ID":"1557cc59-1769-4de0-ba8a-e94c3394b3e3","Type":"ContainerStarted","Data":"eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c"}
Sep 09 03:31:02.571988 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:02.571678    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:02.591984 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:02.591922    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586" podStartSLOduration=2.020683457 podStartE2EDuration="2.591904206s" totalImagesPullingTime="571.220749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:02.590827698 +0000 UTC m=+1511.153154243" watchObservedRunningTime="2026-09-09 03:31:02.591904206 +0000 UTC m=+1511.154230761"
Sep 09 03:31:06.385053 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:06.385014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:08.584673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:08.584635    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:31:10.945996 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:10.945946    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:12.284781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:12.284743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:12.403250 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:12.403207    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:21.094230 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:21.094179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:21.984265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:21.984227    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:24.302553 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:24.302509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:27.498110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:27.498071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:35.356001 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:35.355951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:37.344134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:37.344092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:42.913893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:42.913851    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"]
Sep 09 03:31:42.914482 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:42.914187    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf" podUID="3107598a-5ae3-452a-8fb6-4764aa088008" containerName="maas-api" containerID="cri-o://03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5" gracePeriod=30
Sep 09 03:31:42.940601 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:42.940566    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:43.023625 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.023592    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:43.123197 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.123125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:43.179173 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.179102    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:31:43.202661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.202563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:43.256958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.256926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3107598a-5ae3-452a-8fb6-4764aa088008-openshift-service-ca\") pod \"3107598a-5ae3-452a-8fb6-4764aa088008\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") "
Sep 09 03:31:43.257175 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.256981    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls\") pod \"3107598a-5ae3-452a-8fb6-4764aa088008\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") "
Sep 09 03:31:43.257175 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.257037    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-metrics-certs\") pod \"3107598a-5ae3-452a-8fb6-4764aa088008\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") "
Sep 09 03:31:43.257175 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.257113    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg9ph\" (UniqueName: \"kubernetes.io/projected/3107598a-5ae3-452a-8fb6-4764aa088008-kube-api-access-tg9ph\") pod \"3107598a-5ae3-452a-8fb6-4764aa088008\" (UID: \"3107598a-5ae3-452a-8fb6-4764aa088008\") "
Sep 09 03:31:43.257382 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.257353    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3107598a-5ae3-452a-8fb6-4764aa088008-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3107598a-5ae3-452a-8fb6-4764aa088008" (UID: "3107598a-5ae3-452a-8fb6-4764aa088008"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:43.257632 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.257612    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3107598a-5ae3-452a-8fb6-4764aa088008-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:43.259441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.259410    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3107598a-5ae3-452a-8fb6-4764aa088008" (UID: "3107598a-5ae3-452a-8fb6-4764aa088008"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:31:43.259914 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.259884    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3107598a-5ae3-452a-8fb6-4764aa088008" (UID: "3107598a-5ae3-452a-8fb6-4764aa088008"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:31:43.260018 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.259914    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3107598a-5ae3-452a-8fb6-4764aa088008-kube-api-access-tg9ph" (OuterVolumeSpecName: "kube-api-access-tg9ph") pod "3107598a-5ae3-452a-8fb6-4764aa088008" (UID: "3107598a-5ae3-452a-8fb6-4764aa088008"). InnerVolumeSpecName "kube-api-access-tg9ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:31:43.358661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.358625    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:43.358661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.358653    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3107598a-5ae3-452a-8fb6-4764aa088008-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:43.358661 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.358663    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg9ph\" (UniqueName: \"kubernetes.io/projected/3107598a-5ae3-452a-8fb6-4764aa088008-kube-api-access-tg9ph\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:43.491834 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.491733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:43.848843 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.848808    2562 generic.go:356] "Generic (PLEG): container finished" podID="3107598a-5ae3-452a-8fb6-4764aa088008" containerID="03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5" exitCode=0
Sep 09 03:31:43.849055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.848879    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"
Sep 09 03:31:43.849055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.848902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf" event={"ID":"3107598a-5ae3-452a-8fb6-4764aa088008","Type":"ContainerDied","Data":"03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5"}
Sep 09 03:31:43.849055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.848969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf" event={"ID":"3107598a-5ae3-452a-8fb6-4764aa088008","Type":"ContainerDied","Data":"f25d7804750478a71eab1263b4c34a4aab45263d5f6747ec816ec4b23bdb212e"}
Sep 09 03:31:43.849055 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.848996    2562 scope.go:117] "RemoveContainer" containerID="03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5"
Sep 09 03:31:43.861527 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.861496    2562 scope.go:117] "RemoveContainer" containerID="03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5"
Sep 09 03:31:43.861953 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:31:43.861930    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5\": container with ID starting with 03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5 not found: ID does not exist" containerID="03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5"
Sep 09 03:31:43.862030 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.861964    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5"} err="failed to get container status \"03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5\": rpc error: code = NotFound desc = could not find container \"03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5\": container with ID starting with 03cdcee4da5e721e59a67bb26197ac577a371f1ef4990e3ae80101b0a1a9a6c5 not found: ID does not exist"
Sep 09 03:31:43.873045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.873013    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"]
Sep 09 03:31:43.876406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.876382    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec5029-7b97c77849-jdkjf"]
Sep 09 03:31:43.939092 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:43.939060    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3107598a-5ae3-452a-8fb6-4764aa088008" path="/var/lib/kubelet/pods/3107598a-5ae3-452a-8fb6-4764aa088008/volumes"
Sep 09 03:31:49.330953 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:49.330918    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:49.445441 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:49.445392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:49.622406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:49.622306    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"]
Sep 09 03:31:49.622655 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:49.622625    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy" containerID="cri-o://495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e" gracePeriod=30
Sep 09 03:31:49.648530 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:49.648498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:50.070267 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:50.070228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:50.833010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:50.832957    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:51.694033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:51.693993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:54.884506 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.884434    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:31:54.923940 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.923896    2562 generic.go:356] "Generic (PLEG): container finished" podID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerID="495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e" exitCode=0
Sep 09 03:31:54.924096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.923941    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"
Sep 09 03:31:54.924096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.923960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" event={"ID":"70fa7df2-d9c2-4bcb-897b-985c29cfb716","Type":"ContainerDied","Data":"495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e"}
Sep 09 03:31:54.924096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.924024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk" event={"ID":"70fa7df2-d9c2-4bcb-897b-985c29cfb716","Type":"ContainerDied","Data":"2175aec34ddd9dc060222b6268c6912b36dbbe881f127368836d9977083c1504"}
Sep 09 03:31:54.924096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.924056    2562 scope.go:117] "RemoveContainer" containerID="495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e"
Sep 09 03:31:54.938316 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.938286    2562 scope.go:117] "RemoveContainer" containerID="6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716"
Sep 09 03:31:54.950311 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.950289    2562 scope.go:117] "RemoveContainer" containerID="495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e"
Sep 09 03:31:54.950629 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:31:54.950601    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e\": container with ID starting with 495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e not found: ID does not exist" containerID="495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e"
Sep 09 03:31:54.950750 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.950639    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e"} err="failed to get container status \"495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e\": rpc error: code = NotFound desc = could not find container \"495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e\": container with ID starting with 495ffade9cc5637765dd32d50ab34f0c4d494a844d976fdca8642bf53372c20e not found: ID does not exist"
Sep 09 03:31:54.950750 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.950664    2562 scope.go:117] "RemoveContainer" containerID="6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716"
Sep 09 03:31:54.950982 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:31:54.950967    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716\": container with ID starting with 6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716 not found: ID does not exist" containerID="6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716"
Sep 09 03:31:54.951026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.950989    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716"} err="failed to get container status \"6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716\": rpc error: code = NotFound desc = could not find container \"6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716\": container with ID starting with 6c25501eaeed55c288decaf93690c59a4560f169ae34e1aef9b93459a7f1b716 not found: ID does not exist"
Sep 09 03:31:54.983044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.982993    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istiod-ca-cert\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983062    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-envoy\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983095    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-data\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983168    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4496\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-kube-api-access-n4496\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983221    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-token\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983248    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-podinfo\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983273    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-socket\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983303    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-certs\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983325    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-credential-socket\") pod \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\" (UID: \"70fa7df2-d9c2-4bcb-897b-985c29cfb716\") "
Sep 09 03:31:54.983676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983561    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-data" (OuterVolumeSpecName: "istio-data") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:54.983676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983605    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:54.983829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:54.983829 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983783    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:54.983968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.983835    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:54.984139 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.984118    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:54.984383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.984144    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:54.984383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.984159    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:54.984383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.984201    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:54.984383 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.984215    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:54.985656 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.985632    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:54.985752 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.985658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-kube-api-access-n4496" (OuterVolumeSpecName: "kube-api-access-n4496") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "kube-api-access-n4496". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:31:54.985752 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.985656    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:31:54.985851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:54.985826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-token" (OuterVolumeSpecName: "istio-token") pod "70fa7df2-d9c2-4bcb-897b-985c29cfb716" (UID: "70fa7df2-d9c2-4bcb-897b-985c29cfb716"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:31:55.085513 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:55.085462    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:55.085513 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:55.085509    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4496\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-kube-api-access-n4496\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:55.085513 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:55.085519    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:55.085513 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:55.085527    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fa7df2-d9c2-4bcb-897b-985c29cfb716-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:31:55.248311 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:55.248276    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"]
Sep 09 03:31:55.252558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:55.252518    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ec5029-openshift-default-8d687896d-k2dhk"]
Sep 09 03:31:55.939025 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:31:55.938995    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" path="/var/lib/kubelet/pods/70fa7df2-d9c2-4bcb-897b-985c29cfb716/volumes"
Sep 09 03:32:02.058368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.058334    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7604-c3a3cd"]
Sep 09 03:32:02.058828 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.058811    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy"
Sep 09 03:32:02.058873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.058830    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy"
Sep 09 03:32:02.058873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.058846    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3107598a-5ae3-452a-8fb6-4764aa088008" containerName="maas-api"
Sep 09 03:32:02.058873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.058851    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3107598a-5ae3-452a-8fb6-4764aa088008" containerName="maas-api"
Sep 09 03:32:02.058973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.058925    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3107598a-5ae3-452a-8fb6-4764aa088008" containerName="maas-api"
Sep 09 03:32:02.058973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.058937    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy"
Sep 09 03:32:02.058973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.058943    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy"
Sep 09 03:32:02.063280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.063260    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-c3a3cd"
Sep 09 03:32:02.065640 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.065609    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-49nx9\""
Sep 09 03:32:02.067575 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.067528    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7604-c3a3cd"]
Sep 09 03:32:02.155861 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.155819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznhm\" (UniqueName: \"kubernetes.io/projected/213d8f62-d4b5-44e5-9308-86faedec6186-kube-api-access-wznhm\") pod \"mt-curl-7604-c3a3cd\" (UID: \"213d8f62-d4b5-44e5-9308-86faedec6186\") " pod="opendatahub/mt-curl-7604-c3a3cd"
Sep 09 03:32:02.257164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.257115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznhm\" (UniqueName: \"kubernetes.io/projected/213d8f62-d4b5-44e5-9308-86faedec6186-kube-api-access-wznhm\") pod \"mt-curl-7604-c3a3cd\" (UID: \"213d8f62-d4b5-44e5-9308-86faedec6186\") " pod="opendatahub/mt-curl-7604-c3a3cd"
Sep 09 03:32:02.265971 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.265941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznhm\" (UniqueName: \"kubernetes.io/projected/213d8f62-d4b5-44e5-9308-86faedec6186-kube-api-access-wznhm\") pod \"mt-curl-7604-c3a3cd\" (UID: \"213d8f62-d4b5-44e5-9308-86faedec6186\") " pod="opendatahub/mt-curl-7604-c3a3cd"
Sep 09 03:32:02.375415 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.375320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-c3a3cd"
Sep 09 03:32:02.653958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.653929    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7604-c3a3cd"]
Sep 09 03:32:02.655679 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:32:02.655606    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod213d8f62_d4b5_44e5_9308_86faedec6186.slice/crio-e0cbdb93b428db39d5c18a97a439747b3141e54a037fcfb171321eb1b5a69cb4 WatchSource:0}: Error finding container e0cbdb93b428db39d5c18a97a439747b3141e54a037fcfb171321eb1b5a69cb4: Status 404 returned error can't find the container with id e0cbdb93b428db39d5c18a97a439747b3141e54a037fcfb171321eb1b5a69cb4
Sep 09 03:32:02.975582 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:02.975533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7604-c3a3cd" event={"ID":"213d8f62-d4b5-44e5-9308-86faedec6186","Type":"ContainerStarted","Data":"e0cbdb93b428db39d5c18a97a439747b3141e54a037fcfb171321eb1b5a69cb4"}
Sep 09 03:32:03.983176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:03.983143    2562 generic.go:356] "Generic (PLEG): container finished" podID="213d8f62-d4b5-44e5-9308-86faedec6186" containerID="8162b3a9d87258cda1d614a2e24724d542f37c5b1166fd4fa05a0b39775939a4" exitCode=0
Sep 09 03:32:03.983644 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:03.983266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7604-c3a3cd" event={"ID":"213d8f62-d4b5-44e5-9308-86faedec6186","Type":"ContainerDied","Data":"8162b3a9d87258cda1d614a2e24724d542f37c5b1166fd4fa05a0b39775939a4"}
Sep 09 03:32:05.126065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.126033    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-c3a3cd"
Sep 09 03:32:05.145720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.145644    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7604-c3a3cd_213d8f62-d4b5-44e5-9308-86faedec6186/mt-curl-7604-c3a3cd/0.log"
Sep 09 03:32:05.168360 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.168324    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7604-c3a3cd"]
Sep 09 03:32:05.170894 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.170872    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7604-c3a3cd"]
Sep 09 03:32:05.286895 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.286856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wznhm\" (UniqueName: \"kubernetes.io/projected/213d8f62-d4b5-44e5-9308-86faedec6186-kube-api-access-wznhm\") pod \"213d8f62-d4b5-44e5-9308-86faedec6186\" (UID: \"213d8f62-d4b5-44e5-9308-86faedec6186\") "
Sep 09 03:32:05.289368 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.289335    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213d8f62-d4b5-44e5-9308-86faedec6186-kube-api-access-wznhm" (OuterVolumeSpecName: "kube-api-access-wznhm") pod "213d8f62-d4b5-44e5-9308-86faedec6186" (UID: "213d8f62-d4b5-44e5-9308-86faedec6186"). InnerVolumeSpecName "kube-api-access-wznhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:05.290424 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.290403    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7604-2ba7bd"]
Sep 09 03:32:05.290863 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.290850    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213d8f62-d4b5-44e5-9308-86faedec6186" containerName="mt-curl-7604-c3a3cd"
Sep 09 03:32:05.290900 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.290864    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="213d8f62-d4b5-44e5-9308-86faedec6186" containerName="mt-curl-7604-c3a3cd"
Sep 09 03:32:05.290900 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.290879    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy"
Sep 09 03:32:05.290900 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.290884    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70fa7df2-d9c2-4bcb-897b-985c29cfb716" containerName="istio-proxy"
Sep 09 03:32:05.291004 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.290959    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="213d8f62-d4b5-44e5-9308-86faedec6186" containerName="mt-curl-7604-c3a3cd"
Sep 09 03:32:05.294114 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.294098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-2ba7bd"
Sep 09 03:32:05.305849 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.305821    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7604-2ba7bd"]
Sep 09 03:32:05.388272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.388232    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznhm\" (UniqueName: \"kubernetes.io/projected/213d8f62-d4b5-44e5-9308-86faedec6186-kube-api-access-wznhm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.489334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.489299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmtp\" (UniqueName: \"kubernetes.io/projected/bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726-kube-api-access-sjmtp\") pod \"mt-curl-7604-2ba7bd\" (UID: \"bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726\") " pod="opendatahub/mt-curl-7604-2ba7bd"
Sep 09 03:32:05.563677 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.563577    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"]
Sep 09 03:32:05.563856 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.563833    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586" podUID="1557cc59-1769-4de0-ba8a-e94c3394b3e3" containerName="maas-api" containerID="cri-o://eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c" gracePeriod=30
Sep 09 03:32:05.589456 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.589424    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:05.589941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.589916    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmtp\" (UniqueName: \"kubernetes.io/projected/bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726-kube-api-access-sjmtp\") pod \"mt-curl-7604-2ba7bd\" (UID: \"bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726\") " pod="opendatahub/mt-curl-7604-2ba7bd"
Sep 09 03:32:05.602495 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.602463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmtp\" (UniqueName: \"kubernetes.io/projected/bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726-kube-api-access-sjmtp\") pod \"mt-curl-7604-2ba7bd\" (UID: \"bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726\") " pod="opendatahub/mt-curl-7604-2ba7bd"
Sep 09 03:32:05.607282 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.607256    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-2ba7bd"
Sep 09 03:32:05.694075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.694003    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:05.827515 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.827475    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:32:05.938363 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:32:05.938335    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbddaf5e6_b4f9_4b43_8125_0c1cbc3ac726.slice/crio-6e8014b8d796515d2070d6e1ce52011d93992838923a15873d9abba7bcbb3694 WatchSource:0}: Error finding container 6e8014b8d796515d2070d6e1ce52011d93992838923a15873d9abba7bcbb3694: Status 404 returned error can't find the container with id 6e8014b8d796515d2070d6e1ce52011d93992838923a15873d9abba7bcbb3694
Sep 09 03:32:05.940000 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.939975    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213d8f62-d4b5-44e5-9308-86faedec6186" path="/var/lib/kubelet/pods/213d8f62-d4b5-44e5-9308-86faedec6186/volumes"
Sep 09 03:32:05.942378 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.942358    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7604-2ba7bd"]
Sep 09 03:32:05.948180 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.948146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:05.996788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.995370    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-maas-api-tls\") pod \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") "
Sep 09 03:32:05.996788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.995434    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1557cc59-1769-4de0-ba8a-e94c3394b3e3-openshift-service-ca\") pod \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") "
Sep 09 03:32:05.996788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.995848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1557cc59-1769-4de0-ba8a-e94c3394b3e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1557cc59-1769-4de0-ba8a-e94c3394b3e3" (UID: "1557cc59-1769-4de0-ba8a-e94c3394b3e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:05.996788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.995627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxn7v\" (UniqueName: \"kubernetes.io/projected/1557cc59-1769-4de0-ba8a-e94c3394b3e3-kube-api-access-fxn7v\") pod \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") "
Sep 09 03:32:05.996788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.996405    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-metrics-certs\") pod \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\" (UID: \"1557cc59-1769-4de0-ba8a-e94c3394b3e3\") "
Sep 09 03:32:05.999338 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:05.999304    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1557cc59-1769-4de0-ba8a-e94c3394b3e3-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:06.000527 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.000489    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1557cc59-1769-4de0-ba8a-e94c3394b3e3-kube-api-access-fxn7v" (OuterVolumeSpecName: "kube-api-access-fxn7v") pod "1557cc59-1769-4de0-ba8a-e94c3394b3e3" (UID: "1557cc59-1769-4de0-ba8a-e94c3394b3e3"). InnerVolumeSpecName "kube-api-access-fxn7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:06.001754 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.001729    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1557cc59-1769-4de0-ba8a-e94c3394b3e3" (UID: "1557cc59-1769-4de0-ba8a-e94c3394b3e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:06.002012 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.001982    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1557cc59-1769-4de0-ba8a-e94c3394b3e3" (UID: "1557cc59-1769-4de0-ba8a-e94c3394b3e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:06.005448 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.005427    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-c3a3cd"
Sep 09 03:32:06.005448 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.005441    2562 scope.go:117] "RemoveContainer" containerID="8162b3a9d87258cda1d614a2e24724d542f37c5b1166fd4fa05a0b39775939a4"
Sep 09 03:32:06.006854 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.006833    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7604-2ba7bd" event={"ID":"bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726","Type":"ContainerStarted","Data":"6e8014b8d796515d2070d6e1ce52011d93992838923a15873d9abba7bcbb3694"}
Sep 09 03:32:06.008687 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.008667    2562 generic.go:356] "Generic (PLEG): container finished" podID="1557cc59-1769-4de0-ba8a-e94c3394b3e3" containerID="eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c" exitCode=0
Sep 09 03:32:06.008769 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.008704    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"
Sep 09 03:32:06.008769 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.008754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586" event={"ID":"1557cc59-1769-4de0-ba8a-e94c3394b3e3","Type":"ContainerDied","Data":"eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c"}
Sep 09 03:32:06.008852 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.008798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586" event={"ID":"1557cc59-1769-4de0-ba8a-e94c3394b3e3","Type":"ContainerDied","Data":"6a9a2bde671b8742b8b256e46cf6c315653066e7b468d199cba56cf1671dcc87"}
Sep 09 03:32:06.014955 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.014935    2562 scope.go:117] "RemoveContainer" containerID="eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c"
Sep 09 03:32:06.024926 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.024904    2562 scope.go:117] "RemoveContainer" containerID="eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c"
Sep 09 03:32:06.025259 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:32:06.025236    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c\": container with ID starting with eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c not found: ID does not exist" containerID="eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c"
Sep 09 03:32:06.025340 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.025272    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c"} err="failed to get container status \"eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c\": rpc error: code = NotFound desc = could not find container \"eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c\": container with ID starting with eec2c58e11b2842149ad0a3f44c7a27bea16d761c604fd8c6220050f468d710c not found: ID does not exist"
Sep 09 03:32:06.051573 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.051527    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"]
Sep 09 03:32:06.073033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.072965    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-811c96-6958dd7cf6-94586"]
Sep 09 03:32:06.100289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.100262    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:06.100289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.100291    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxn7v\" (UniqueName: \"kubernetes.io/projected/1557cc59-1769-4de0-ba8a-e94c3394b3e3-kube-api-access-fxn7v\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:06.100469 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.100302    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1557cc59-1769-4de0-ba8a-e94c3394b3e3-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:06.123437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.123408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:06.335685 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:06.335653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:07.019276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:07.019238    2562 generic.go:356] "Generic (PLEG): container finished" podID="bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726" containerID="f72978d0e1417568ed6454af9283a2ecba120339da242f6385f4e4ada6f2c339" exitCode=0
Sep 09 03:32:07.019470 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:07.019302    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7604-2ba7bd" event={"ID":"bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726","Type":"ContainerDied","Data":"f72978d0e1417568ed6454af9283a2ecba120339da242f6385f4e4ada6f2c339"}
Sep 09 03:32:07.938779 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:07.938738    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1557cc59-1769-4de0-ba8a-e94c3394b3e3" path="/var/lib/kubelet/pods/1557cc59-1769-4de0-ba8a-e94c3394b3e3/volumes"
Sep 09 03:32:08.171068 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:08.171041    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-2ba7bd"
Sep 09 03:32:08.193291 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:08.193202    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7604-2ba7bd_bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726/mt-curl-7604-2ba7bd/0.log"
Sep 09 03:32:08.221233 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:08.221201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjmtp\" (UniqueName: \"kubernetes.io/projected/bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726-kube-api-access-sjmtp\") pod \"bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726\" (UID: \"bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726\") "
Sep 09 03:32:08.223444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:08.223419    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7604-2ba7bd"]
Sep 09 03:32:08.224035 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:08.224016    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726-kube-api-access-sjmtp" (OuterVolumeSpecName: "kube-api-access-sjmtp") pod "bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726" (UID: "bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726"). InnerVolumeSpecName "kube-api-access-sjmtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:08.226041 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:08.226021    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7604-2ba7bd"]
Sep 09 03:32:08.322940 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:08.322885    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmtp\" (UniqueName: \"kubernetes.io/projected/bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726-kube-api-access-sjmtp\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:09.032497 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:09.032470    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e8014b8d796515d2070d6e1ce52011d93992838923a15873d9abba7bcbb3694"
Sep 09 03:32:09.033033 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:09.032505    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7604-2ba7bd"
Sep 09 03:32:09.042178 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:32:09.042144    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7604-2ba7bd\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726" pod="opendatahub/mt-curl-7604-2ba7bd"
Sep 09 03:32:09.938699 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:09.938667    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726" path="/var/lib/kubelet/pods/bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726/volumes"
Sep 09 03:32:10.217757 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:10.217659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:11.213224 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:11.213183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:11.994178 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:11.994144    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:12.129129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:12.129090    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:12.194743 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:12.194703    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:12.458114 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:12.458074    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:12.843397 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:12.843341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:13.358785 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:13.358750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:18.453789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.453751    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"]
Sep 09 03:32:18.454259 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.454244    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726" containerName="mt-curl-7604-2ba7bd"
Sep 09 03:32:18.454300 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.454262    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726" containerName="mt-curl-7604-2ba7bd"
Sep 09 03:32:18.454300 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.454291    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1557cc59-1769-4de0-ba8a-e94c3394b3e3" containerName="maas-api"
Sep 09 03:32:18.454300 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.454297    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1557cc59-1769-4de0-ba8a-e94c3394b3e3" containerName="maas-api"
Sep 09 03:32:18.454399 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.454385    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1557cc59-1769-4de0-ba8a-e94c3394b3e3" containerName="maas-api"
Sep 09 03:32:18.454399 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.454396    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bddaf5e6-b4f9-4b43-8125-0c1cbc3ac726" containerName="mt-curl-7604-2ba7bd"
Sep 09 03:32:18.465924 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.465897    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.476553 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.476506    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"]
Sep 09 03:32:18.477711 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.477687    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"kube-root-ca.crt\""
Sep 09 03:32:18.477853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.477727    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"default-dockercfg-xnl9c\""
Sep 09 03:32:18.477853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.477802    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"rate-test-model-1a85fa-kserve-self-signed-certs\""
Sep 09 03:32:18.477853 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.477807    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"openshift-service-ca.crt\""
Sep 09 03:32:18.527623 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.527595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-model-cache\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.527756 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.527672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-dshm\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.527756 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.527718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqw5\" (UniqueName: \"kubernetes.io/projected/f5c0d218-9872-427a-b7fc-7e9122b96a28-kube-api-access-czqw5\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.527845 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.527780    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-tmp-dir\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.527845 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.527799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-home\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.527845 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.527820    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c0d218-9872-427a-b7fc-7e9122b96a28-tls-certs\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.612493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.612444    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:18.628824 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.628790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-tmp-dir\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.628986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.628829    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-home\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.628986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.628852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c0d218-9872-427a-b7fc-7e9122b96a28-tls-certs\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.628986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.628894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-model-cache\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.628986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.628944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-dshm\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.629153 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.629003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqw5\" (UniqueName: \"kubernetes.io/projected/f5c0d218-9872-427a-b7fc-7e9122b96a28-kube-api-access-czqw5\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.629290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.629259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-tmp-dir\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.629370 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.629273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-home\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.629370 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.629319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-model-cache\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.631485 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.631463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-dshm\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.631798 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.631783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c0d218-9872-427a-b7fc-7e9122b96a28-tls-certs\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.643377 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.643339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqw5\" (UniqueName: \"kubernetes.io/projected/f5c0d218-9872-427a-b7fc-7e9122b96a28-kube-api-access-czqw5\") pod \"rate-test-model-1a85fa-kserve-5cb66b747-z5tgt\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:18.775793 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:18.775699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:19.065353 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:19.065327    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"]
Sep 09 03:32:19.066728 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:32:19.066701    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5c0d218_9872_427a_b7fc_7e9122b96a28.slice/crio-9c850ac01038e95bccbeecaab2bc5911de863ac2c5a3df7f0fd146b5f299d1b8 WatchSource:0}: Error finding container 9c850ac01038e95bccbeecaab2bc5911de863ac2c5a3df7f0fd146b5f299d1b8: Status 404 returned error can't find the container with id 9c850ac01038e95bccbeecaab2bc5911de863ac2c5a3df7f0fd146b5f299d1b8
Sep 09 03:32:19.096265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:19.096241    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt" event={"ID":"f5c0d218-9872-427a-b7fc-7e9122b96a28","Type":"ContainerStarted","Data":"9c850ac01038e95bccbeecaab2bc5911de863ac2c5a3df7f0fd146b5f299d1b8"}
Sep 09 03:32:20.103607 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:20.103570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt" event={"ID":"f5c0d218-9872-427a-b7fc-7e9122b96a28","Type":"ContainerStarted","Data":"001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe"}
Sep 09 03:32:21.172720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:21.172681    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:23.373689 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.373631    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt" podStartSLOduration=5.373609977 podStartE2EDuration="5.373609977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:32:20.131595649 +0000 UTC m=+1588.693922213" watchObservedRunningTime="2026-09-09 03:32:23.373609977 +0000 UTC m=+1591.935936568"
Sep 09 03:32:23.374799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.374770    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"]
Sep 09 03:32:23.375031 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.375004    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9" podUID="ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" containerName="maas-api" containerID="cri-o://cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43" gracePeriod=30
Sep 09 03:32:23.466911 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.466876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:23.643704 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.643669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:23.651164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.651134    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:32:23.785445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.784188    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-maas-api-tls\") pod \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") "
Sep 09 03:32:23.785445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.784250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-openshift-service-ca\") pod \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") "
Sep 09 03:32:23.785445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.784388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2668t\" (UniqueName: \"kubernetes.io/projected/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-kube-api-access-2668t\") pod \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") "
Sep 09 03:32:23.785445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.784422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-metrics-certs\") pod \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\" (UID: \"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa\") "
Sep 09 03:32:23.788724 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.788366    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" (UID: "ea5d11b3-64e6-4433-8c9b-d2d68401c7aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:23.791451 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.791228    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" (UID: "ea5d11b3-64e6-4433-8c9b-d2d68401c7aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:23.792025 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.791748    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" (UID: "ea5d11b3-64e6-4433-8c9b-d2d68401c7aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:23.794054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.794019    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-kube-api-access-2668t" (OuterVolumeSpecName: "kube-api-access-2668t") pod "ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" (UID: "ea5d11b3-64e6-4433-8c9b-d2d68401c7aa"). InnerVolumeSpecName "kube-api-access-2668t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:23.855029 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.854989    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:23.885803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.885719    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.885803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.885749    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.885803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.885759    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2668t\" (UniqueName: \"kubernetes.io/projected/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-kube-api-access-2668t\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.885803 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.885769    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.976786 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:23.976745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:24.132393 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.132358    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" containerID="cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43" exitCode=0
Sep 09 03:32:24.132614 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.132420    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"
Sep 09 03:32:24.132614 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.132445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9" event={"ID":"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa","Type":"ContainerDied","Data":"cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43"}
Sep 09 03:32:24.132614 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.132512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9" event={"ID":"ea5d11b3-64e6-4433-8c9b-d2d68401c7aa","Type":"ContainerDied","Data":"45a70d1fb65296b0bf168e1dfee8a9d26b1d9fa4a1a2a942ac4c9dbfb7f4a8a7"}
Sep 09 03:32:24.132614 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.132559    2562 scope.go:117] "RemoveContainer" containerID="cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43"
Sep 09 03:32:24.142966 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.142936    2562 scope.go:117] "RemoveContainer" containerID="cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43"
Sep 09 03:32:24.143353 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:32:24.143325    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43\": container with ID starting with cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43 not found: ID does not exist" containerID="cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43"
Sep 09 03:32:24.143455 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.143365    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43"} err="failed to get container status \"cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43\": rpc error: code = NotFound desc = could not find container \"cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43\": container with ID starting with cb7bbd3cf180a1f3a73da769e8bd8c5bb37080cb76b3fd87ea6e4c9ac1ea1c43 not found: ID does not exist"
Sep 09 03:32:24.151332 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.151304    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"]
Sep 09 03:32:24.157363 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.157339    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-207ebd-65f84869cb-tdgt9"]
Sep 09 03:32:24.300045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:24.300011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:25.938907 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:25.938864    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" path="/var/lib/kubelet/pods/ea5d11b3-64e6-4433-8c9b-d2d68401c7aa/volumes"
Sep 09 03:32:28.776589 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:28.776556    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:28.776992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:28.776607    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:28.789133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:28.789103    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:29.186239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:29.186207    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:32:31.646247 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:31.646205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:31.815192 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:31.815156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:31.983468 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:31.983381    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"]
Sep 09 03:32:31.983657 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:31.983615    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy" containerID="cri-o://74bd58ce8b23ce98408c7941f2bf686fece5dc15de5b5daa3b4ab52bda303114" gracePeriod=30
Sep 09 03:32:32.026301 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:32.026264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:32.594101 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:32.594051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:37.224851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.224813    2562 generic.go:356] "Generic (PLEG): container finished" podID="2e85f576-093f-4bb8-85ed-91636531b445" containerID="74bd58ce8b23ce98408c7941f2bf686fece5dc15de5b5daa3b4ab52bda303114" exitCode=0
Sep 09 03:32:37.225214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.224880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" event={"ID":"2e85f576-093f-4bb8-85ed-91636531b445","Type":"ContainerDied","Data":"74bd58ce8b23ce98408c7941f2bf686fece5dc15de5b5daa3b4ab52bda303114"}
Sep 09 03:32:37.225214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.224939    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns" event={"ID":"2e85f576-093f-4bb8-85ed-91636531b445","Type":"ContainerDied","Data":"474fa0af14de87a7eb4d7eb676d24a5d078d8b2520b11a01013ad4f556f58e24"}
Sep 09 03:32:37.225214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.224955    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474fa0af14de87a7eb4d7eb676d24a5d078d8b2520b11a01013ad4f556f58e24"
Sep 09 03:32:37.225214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.224970    2562 scope.go:117] "RemoveContainer" containerID="172e0124e24706435f342348817559733045cedd3cc6112247576d22b0a758bb"
Sep 09 03:32:37.243271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.243223    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:32:37.419297 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419258    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85f576-093f-4bb8-85ed-91636531b445-istio-podinfo\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419297 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85f576-093f-4bb8-85ed-91636531b445-istiod-ca-cert\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419328    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-credential-socket\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419359    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-certs\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419411    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-istio-token\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419453    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-socket\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419479    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-envoy\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419503    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-data\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419567 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419525    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nrkr\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-kube-api-access-8nrkr\") pod \"2e85f576-093f-4bb8-85ed-91636531b445\" (UID: \"2e85f576-093f-4bb8-85ed-91636531b445\") "
Sep 09 03:32:37.419920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419711    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:37.419920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419822    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e85f576-093f-4bb8-85ed-91636531b445-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:37.419920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419883    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:37.420082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.419990    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:37.420082 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.420044    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:37.420164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.420081    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:37.422372 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.422339    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e85f576-093f-4bb8-85ed-91636531b445-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:32:37.422515 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.422375    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:37.422515 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.422459    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:37.422515 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.422472    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-kube-api-access-8nrkr" (OuterVolumeSpecName: "kube-api-access-8nrkr") pod "2e85f576-093f-4bb8-85ed-91636531b445" (UID: "2e85f576-093f-4bb8-85ed-91636531b445"). InnerVolumeSpecName "kube-api-access-8nrkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:37.520771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.520665    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85f576-093f-4bb8-85ed-91636531b445-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:37.520771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.520709    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85f576-093f-4bb8-85ed-91636531b445-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:37.520771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.520719    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:37.520771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.520730    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:37.520771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.520738    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:37.520771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.520745    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:37.520771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.520753    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85f576-093f-4bb8-85ed-91636531b445-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:37.520771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:37.520761    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nrkr\" (UniqueName: \"kubernetes.io/projected/2e85f576-093f-4bb8-85ed-91636531b445-kube-api-access-8nrkr\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:38.232218 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:38.232136    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"
Sep 09 03:32:38.296955 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:38.296918    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"]
Sep 09 03:32:38.305069 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:38.305034    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-207ebd-openshift-default-6bf47b7c78-bx5ns"]
Sep 09 03:32:39.938931 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:39.938900    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e85f576-093f-4bb8-85ed-91636531b445" path="/var/lib/kubelet/pods/2e85f576-093f-4bb8-85ed-91636531b445/volumes"
Sep 09 03:32:43.107227 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.107144    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"]
Sep 09 03:32:43.107619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.107371    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr" podUID="83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" containerName="maas-api" containerID="cri-o://402ee7a4ae80ec6419ea625e72896b1c7bb7ba16bbf0e69d859758d39640b7c1" gracePeriod=30
Sep 09 03:32:43.157422 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.157387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:43.270190 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.270159    2562 generic.go:356] "Generic (PLEG): container finished" podID="83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" containerID="402ee7a4ae80ec6419ea625e72896b1c7bb7ba16bbf0e69d859758d39640b7c1" exitCode=0
Sep 09 03:32:43.270391 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.270221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr" event={"ID":"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc","Type":"ContainerDied","Data":"402ee7a4ae80ec6419ea625e72896b1c7bb7ba16bbf0e69d859758d39640b7c1"}
Sep 09 03:32:43.294231 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.294096    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:43.374433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.374366    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:32:43.389552 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.389507    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:43.476572 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.476510    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p66zz\" (UniqueName: \"kubernetes.io/projected/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-kube-api-access-p66zz\") pod \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") "
Sep 09 03:32:43.476763 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.476629    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-openshift-service-ca\") pod \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") "
Sep 09 03:32:43.476763 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.476661    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-metrics-certs\") pod \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") "
Sep 09 03:32:43.476876 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.476855    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls\") pod \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\" (UID: \"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc\") "
Sep 09 03:32:43.476996 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.476974    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" (UID: "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:43.477204 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.477182    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:43.478963 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.478926    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" (UID: "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:43.479060 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.478978    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" (UID: "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:43.479196 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.479175    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-kube-api-access-p66zz" (OuterVolumeSpecName: "kube-api-access-p66zz") pod "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" (UID: "83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc"). InnerVolumeSpecName "kube-api-access-p66zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:43.549912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.549866    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:43.578606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.578572    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66zz\" (UniqueName: \"kubernetes.io/projected/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-kube-api-access-p66zz\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:43.578606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.578602    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:43.578606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.578615    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:32:43.685249 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.685157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:43.934186 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:43.934144    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:44.278346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:44.278259    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"
Sep 09 03:32:44.278346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:44.278286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr" event={"ID":"83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc","Type":"ContainerDied","Data":"c44bf898265ebad893df10af08d05cc59d7b2b3f7eb63dcc01de2660328d0253"}
Sep 09 03:32:44.278902 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:44.278355    2562 scope.go:117] "RemoveContainer" containerID="402ee7a4ae80ec6419ea625e72896b1c7bb7ba16bbf0e69d859758d39640b7c1"
Sep 09 03:32:44.298464 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:44.298428    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"]
Sep 09 03:32:44.305438 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:44.305408    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4c8420-89bf7c5d6-9m8tr"]
Sep 09 03:32:45.938506 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:45.938469    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" path="/var/lib/kubelet/pods/83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc/volumes"
Sep 09 03:32:50.173101 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:50.173066    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:50.276215 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:50.276172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:50.468688 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:50.468595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:50.983171 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:50.983133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:52.209204 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:52.209167    2562 scope.go:117] "RemoveContainer" containerID="74bd58ce8b23ce98408c7941f2bf686fece5dc15de5b5daa3b4ab52bda303114"
Sep 09 03:32:54.877296 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:32:54.877255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:29.811972 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:29.811926    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"]
Sep 09 03:33:29.851026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:29.850984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:30.916005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:30.915963    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:31.329960 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:31.329911    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:31.639857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:31.639755    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:32.092740 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:32.092712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:35.202768 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:35.202738    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:35.300260 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:35.300219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:36.362466 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.362428    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"]
Sep 09 03:33:36.363140 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363119    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy"
Sep 09 03:33:36.363185 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363145    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy"
Sep 09 03:33:36.363185 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363164    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" containerName="maas-api"
Sep 09 03:33:36.363185 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363172    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" containerName="maas-api"
Sep 09 03:33:36.363279 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363192    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy"
Sep 09 03:33:36.363279 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363200    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy"
Sep 09 03:33:36.363279 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363220    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" containerName="maas-api"
Sep 09 03:33:36.363279 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363228    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" containerName="maas-api"
Sep 09 03:33:36.363404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363337    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy"
Sep 09 03:33:36.363404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363351    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c5aad4-e26c-47a0-8cf9-7c7dbc9b6edc" containerName="maas-api"
Sep 09 03:33:36.363404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.363359    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea5d11b3-64e6-4433-8c9b-d2d68401c7aa" containerName="maas-api"
Sep 09 03:33:36.367011 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.366990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.369582 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.369558    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ed81bb-serving-cert\""
Sep 09 03:33:36.382449 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.382418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"]
Sep 09 03:33:36.408038 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.408005    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:36.494054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.494008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12040f9-2f1d-4b88-8897-3c0c96979bac-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.494325 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.494084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.494325 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.494112    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mm2j\" (UniqueName: \"kubernetes.io/projected/f12040f9-2f1d-4b88-8897-3c0c96979bac-kube-api-access-2mm2j\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.494325 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.494187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.515463 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.515425    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:36.595711 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.595667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.595909 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.595719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mm2j\" (UniqueName: \"kubernetes.io/projected/f12040f9-2f1d-4b88-8897-3c0c96979bac-kube-api-access-2mm2j\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.595909 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.595814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.595909 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.595888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12040f9-2f1d-4b88-8897-3c0c96979bac-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.596535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.596506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12040f9-2f1d-4b88-8897-3c0c96979bac-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.598563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.598525    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.598696 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.598635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.609086 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.609058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mm2j\" (UniqueName: \"kubernetes.io/projected/f12040f9-2f1d-4b88-8897-3c0c96979bac-kube-api-access-2mm2j\") pod \"maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.678359 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.678258    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:36.961333 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:36.961297    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"]
Sep 09 03:33:36.962281 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:33:36.962255    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12040f9_2f1d_4b88_8897_3c0c96979bac.slice/crio-cb7658d419da8f2617bdd1858e1681fda0da8a9c873374cb256260283b09c0ad WatchSource:0}: Error finding container cb7658d419da8f2617bdd1858e1681fda0da8a9c873374cb256260283b09c0ad: Status 404 returned error can't find the container with id cb7658d419da8f2617bdd1858e1681fda0da8a9c873374cb256260283b09c0ad
Sep 09 03:33:37.099912 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:37.099878    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:37.254864 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:37.254772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:37.622982 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:37.622935    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6" event={"ID":"f12040f9-2f1d-4b88-8897-3c0c96979bac","Type":"ContainerStarted","Data":"683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5"}
Sep 09 03:33:37.623475 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:37.623000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6" event={"ID":"f12040f9-2f1d-4b88-8897-3c0c96979bac","Type":"ContainerStarted","Data":"cb7658d419da8f2617bdd1858e1681fda0da8a9c873374cb256260283b09c0ad"}
Sep 09 03:33:37.623475 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:37.623024    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:37.647064 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:37.646999    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6" podStartSLOduration=1.287422498 podStartE2EDuration="1.646979736s" totalImagesPullingTime="359.557238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:33:37.646026159 +0000 UTC m=+1666.208352705" watchObservedRunningTime="2026-09-09 03:33:37.646979736 +0000 UTC m=+1666.209306280"
Sep 09 03:33:43.635615 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:43.635576    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:33:44.833493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:44.833455    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt" podUID="f5c0d218-9872-427a-b7fc-7e9122b96a28" containerName="main" containerID="cri-o://001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe" gracePeriod=45
Sep 09 03:33:45.093086 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.093019    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:33:45.186237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186203    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c0d218-9872-427a-b7fc-7e9122b96a28-tls-certs\") pod \"f5c0d218-9872-427a-b7fc-7e9122b96a28\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") "
Sep 09 03:33:45.186237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-dshm\") pod \"f5c0d218-9872-427a-b7fc-7e9122b96a28\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") "
Sep 09 03:33:45.186514 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186302    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-home\") pod \"f5c0d218-9872-427a-b7fc-7e9122b96a28\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") "
Sep 09 03:33:45.186514 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186335    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czqw5\" (UniqueName: \"kubernetes.io/projected/f5c0d218-9872-427a-b7fc-7e9122b96a28-kube-api-access-czqw5\") pod \"f5c0d218-9872-427a-b7fc-7e9122b96a28\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") "
Sep 09 03:33:45.186514 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186363    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-model-cache\") pod \"f5c0d218-9872-427a-b7fc-7e9122b96a28\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") "
Sep 09 03:33:45.186514 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-tmp-dir\") pod \"f5c0d218-9872-427a-b7fc-7e9122b96a28\" (UID: \"f5c0d218-9872-427a-b7fc-7e9122b96a28\") "
Sep 09 03:33:45.186774 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-home" (OuterVolumeSpecName: "home") pod "f5c0d218-9872-427a-b7fc-7e9122b96a28" (UID: "f5c0d218-9872-427a-b7fc-7e9122b96a28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:45.186774 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186679    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5c0d218-9872-427a-b7fc-7e9122b96a28" (UID: "f5c0d218-9872-427a-b7fc-7e9122b96a28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:45.186774 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186750    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5c0d218-9872-427a-b7fc-7e9122b96a28" (UID: "f5c0d218-9872-427a-b7fc-7e9122b96a28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:45.186919 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186825    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:33:45.186919 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.186842    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:33:45.188813 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.188783    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c0d218-9872-427a-b7fc-7e9122b96a28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5c0d218-9872-427a-b7fc-7e9122b96a28" (UID: "f5c0d218-9872-427a-b7fc-7e9122b96a28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:33:45.188932 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.188838    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-dshm" (OuterVolumeSpecName: "dshm") pod "f5c0d218-9872-427a-b7fc-7e9122b96a28" (UID: "f5c0d218-9872-427a-b7fc-7e9122b96a28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:45.188932 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.188838    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c0d218-9872-427a-b7fc-7e9122b96a28-kube-api-access-czqw5" (OuterVolumeSpecName: "kube-api-access-czqw5") pod "f5c0d218-9872-427a-b7fc-7e9122b96a28" (UID: "f5c0d218-9872-427a-b7fc-7e9122b96a28"). InnerVolumeSpecName "kube-api-access-czqw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:33:45.288274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.288235    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c0d218-9872-427a-b7fc-7e9122b96a28-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:33:45.288274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.288266    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:33:45.288274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.288275    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czqw5\" (UniqueName: \"kubernetes.io/projected/f5c0d218-9872-427a-b7fc-7e9122b96a28-kube-api-access-czqw5\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:33:45.288274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.288283    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c0d218-9872-427a-b7fc-7e9122b96a28-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:33:45.683517 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.683480    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5c0d218-9872-427a-b7fc-7e9122b96a28" containerID="001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe" exitCode=0
Sep 09 03:33:45.683737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.683524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt" event={"ID":"f5c0d218-9872-427a-b7fc-7e9122b96a28","Type":"ContainerDied","Data":"001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe"}
Sep 09 03:33:45.683737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.683576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt" event={"ID":"f5c0d218-9872-427a-b7fc-7e9122b96a28","Type":"ContainerDied","Data":"9c850ac01038e95bccbeecaab2bc5911de863ac2c5a3df7f0fd146b5f299d1b8"}
Sep 09 03:33:45.683737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.683594    2562 scope.go:117] "RemoveContainer" containerID="001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe"
Sep 09 03:33:45.683737 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.683526    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"
Sep 09 03:33:45.695805 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.695784    2562 scope.go:117] "RemoveContainer" containerID="001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe"
Sep 09 03:33:45.696180 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:33:45.696160    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe\": container with ID starting with 001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe not found: ID does not exist" containerID="001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe"
Sep 09 03:33:45.696219 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.696192    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe"} err="failed to get container status \"001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe\": rpc error: code = NotFound desc = could not find container \"001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe\": container with ID starting with 001cf53cf9e02ee477490fa33dc009ec74af959ecd566e52465cc5c53224a4fe not found: ID does not exist"
Sep 09 03:33:45.717666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.717621    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"]
Sep 09 03:33:45.720276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.720247    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/rate-test-model-1a85fa-kserve-5cb66b747-z5tgt"]
Sep 09 03:33:45.939928 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:33:45.939841    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5c0d218-9872-427a-b7fc-7e9122b96a28" path="/var/lib/kubelet/pods/f5c0d218-9872-427a-b7fc-7e9122b96a28/volumes"
Sep 09 03:34:00.179346 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:00.179304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:00.868799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:00.868768    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:01.212215 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.212116    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"]
Sep 09 03:34:01.212673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.212578    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c0d218-9872-427a-b7fc-7e9122b96a28" containerName="main"
Sep 09 03:34:01.212673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.212592    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c0d218-9872-427a-b7fc-7e9122b96a28" containerName="main"
Sep 09 03:34:01.212673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.212656    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e85f576-093f-4bb8-85ed-91636531b445" containerName="istio-proxy"
Sep 09 03:34:01.212673 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.212666    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c0d218-9872-427a-b7fc-7e9122b96a28" containerName="main"
Sep 09 03:34:01.217257 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.217235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.220129 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.220103    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-aacd65-openshift-default-dockercfg-5f2zx\""
Sep 09 03:34:01.234290 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.234247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"]
Sep 09 03:34:01.243911 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.243878    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:01.244181 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-certs\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.244342 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-envoy\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.244478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-podinfo\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.244478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-data\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.244478 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-credential-socket\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.244720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-socket\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.244720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244646    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqhz\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-kube-api-access-5nqhz\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.244720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istiod-ca-cert\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.244720 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.244697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-token\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.327953 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.327917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:01.346145 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-credential-socket\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-socket\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqhz\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-kube-api-access-5nqhz\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istiod-ca-cert\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-token\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-certs\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-envoy\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-podinfo\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-data\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346648    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-credential-socket\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346659    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-certs\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-data\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.346920 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.346893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-socket\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.347143 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.347112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istiod-ca-cert\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.349697 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.349674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-envoy\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.349946 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.349924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-podinfo\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.356166 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.356140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqhz\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-kube-api-access-5nqhz\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.356279 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.356148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-token\") pod \"e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") " pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.529030 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.528933    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:01.812513 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.812483    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"]
Sep 09 03:34:01.814168 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:34:01.814136    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bdddde_bd1c_4033_a7b2_fca98c7ae17d.slice/crio-3060adb194bc9b8b500ef4d7c35cb54b426ff17d82ab53d04dda4cc7f38468e5 WatchSource:0}: Error finding container 3060adb194bc9b8b500ef4d7c35cb54b426ff17d82ab53d04dda4cc7f38468e5: Status 404 returned error can't find the container with id 3060adb194bc9b8b500ef4d7c35cb54b426ff17d82ab53d04dda4cc7f38468e5
Sep 09 03:34:01.816705 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.816670    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:34:01.816800 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.816742    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:34:01.816800 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:01.816769    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:34:02.786535 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:02.786495    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" event={"ID":"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d","Type":"ContainerStarted","Data":"c4187379f8fc6b31843758eb056a4f66bc219d95e6868c47877e8c786870ddb5"}
Sep 09 03:34:02.787045 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:02.786591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" event={"ID":"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d","Type":"ContainerStarted","Data":"3060adb194bc9b8b500ef4d7c35cb54b426ff17d82ab53d04dda4cc7f38468e5"}
Sep 09 03:34:02.807269 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:02.807211    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" podStartSLOduration=1.807194204 podStartE2EDuration="1.807194204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:34:02.805765293 +0000 UTC m=+1691.368091829" watchObservedRunningTime="2026-09-09 03:34:02.807194204 +0000 UTC m=+1691.369520749"
Sep 09 03:34:03.529370 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:03.529324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:03.534901 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:03.534870    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:03.792608 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:03.792572    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:03.793582 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:03.793530    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:04.725571 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:04.725500    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:04.937508 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:04.937473    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:05.875202 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.875162    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"]
Sep 09 03:34:05.879462 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.879434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:05.884557 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.884245    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-aacd65-serving-cert\""
Sep 09 03:34:05.897316 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.897278    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"]
Sep 09 03:34:05.925892 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.925856    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:05.993181 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.993124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84jkc\" (UniqueName: \"kubernetes.io/projected/90657ced-fd95-4bef-ba46-879d9fa682ae-kube-api-access-84jkc\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:05.993941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.993907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:05.994100 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.994033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:05.994202 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:05.994177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90657ced-fd95-4bef-ba46-879d9fa682ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.050831 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.050794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:06.095281 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.095239    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84jkc\" (UniqueName: \"kubernetes.io/projected/90657ced-fd95-4bef-ba46-879d9fa682ae-kube-api-access-84jkc\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.095493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.095302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.095493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.095385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.095493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.095450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90657ced-fd95-4bef-ba46-879d9fa682ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.095729 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:34:06.095510    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-serving-cert: secret "maas-api-e2e-ipp-b-aacd65-serving-cert" not found
Sep 09 03:34:06.095729 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:34:06.095612    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls podName:90657ced-fd95-4bef-ba46-879d9fa682ae nodeName:}" failed. No retries permitted until 2026-09-09 03:34:06.595589377 +0000 UTC m=+1695.157915917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls") pod "maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz" (UID: "90657ced-fd95-4bef-ba46-879d9fa682ae") : secret "maas-api-e2e-ipp-b-aacd65-serving-cert" not found
Sep 09 03:34:06.096269 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.096244    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90657ced-fd95-4bef-ba46-879d9fa682ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.098362 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.098338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.104699 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.104673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84jkc\" (UniqueName: \"kubernetes.io/projected/90657ced-fd95-4bef-ba46-879d9fa682ae-kube-api-access-84jkc\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.230254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.230151    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:06.538353 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.538320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:06.600138 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.600107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.602857 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.602826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:06.643063 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.642974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:06.794320 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:06.793951    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:07.085921 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:07.085896    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"]
Sep 09 03:34:07.087712 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:34:07.087685    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90657ced_fd95_4bef_ba46_879d9fa682ae.slice/crio-b9d275c3938be5edf5405b41d781ab62acf1e1c5f8d3ee2e4e660088710db6c3 WatchSource:0}: Error finding container b9d275c3938be5edf5405b41d781ab62acf1e1c5f8d3ee2e4e660088710db6c3: Status 404 returned error can't find the container with id b9d275c3938be5edf5405b41d781ab62acf1e1c5f8d3ee2e4e660088710db6c3
Sep 09 03:34:07.820477 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:07.820443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz" event={"ID":"90657ced-fd95-4bef-ba46-879d9fa682ae","Type":"ContainerStarted","Data":"3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f"}
Sep 09 03:34:07.820736 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:07.820505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz" event={"ID":"90657ced-fd95-4bef-ba46-879d9fa682ae","Type":"ContainerStarted","Data":"b9d275c3938be5edf5405b41d781ab62acf1e1c5f8d3ee2e4e660088710db6c3"}
Sep 09 03:34:07.820736 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:07.820573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:07.849940 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:07.849870    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz" podStartSLOduration=2.437975436 podStartE2EDuration="2.849843813s" totalImagesPullingTime="411.868377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:34:07.848429102 +0000 UTC m=+1696.410755646" watchObservedRunningTime="2026-09-09 03:34:07.849843813 +0000 UTC m=+1696.412170358"
Sep 09 03:34:11.850809 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:11.850765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:13.833155 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:13.833072    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:34:22.921882 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:22.921849    2562 generic.go:356] "Generic (PLEG): container finished" podID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerID="c4187379f8fc6b31843758eb056a4f66bc219d95e6868c47877e8c786870ddb5" exitCode=0
Sep 09 03:34:22.922312 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:22.921923    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" event={"ID":"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d","Type":"ContainerDied","Data":"c4187379f8fc6b31843758eb056a4f66bc219d95e6868c47877e8c786870ddb5"}
Sep 09 03:34:22.922417 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:22.922399    2562 scope.go:117] "RemoveContainer" containerID="c4187379f8fc6b31843758eb056a4f66bc219d95e6868c47877e8c786870ddb5"
Sep 09 03:34:22.923580 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:22.923523    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:34:22.923648 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:22.923632    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:34:22.923689 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:22.923666    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:34:23.929406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:23.929372    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" event={"ID":"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d","Type":"ContainerStarted","Data":"ed02db8694c12662e52a0d994c9d951ae2158da511a92e466f8869c942a0d078"}
Sep 09 03:34:24.529256 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:24.529220    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:24.529481 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:24.529459    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 03:34:24.529529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:24.529511    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 03:34:25.529871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:25.529834    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 03:34:25.530287 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:25.529906    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 03:34:26.529305 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:26.529263    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 03:34:26.529555 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:26.529322    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 03:34:27.529447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:27.529406    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 03:34:27.529899 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:27.529470    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 03:34:28.529527 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:28.529490    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 03:34:28.529982 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:28.529591    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 03:34:29.534687 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:29.534651    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:29.535097 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:29.534956    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:29.535745 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:29.535726    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:34:29.805696 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:29.805605    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:40.312785 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.312741    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"]
Sep 09 03:34:40.317666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.317642    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.320058 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.320031    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ed81bb\"/\"openshift-service-ca.crt\""
Sep 09 03:34:40.321319 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.321290    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ed81bb\"/\"kube-root-ca.crt\""
Sep 09 03:34:40.321879 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.321858    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ed81bb\"/\"ipp-route-ed81bb-kserve-self-signed-certs\""
Sep 09 03:34:40.322340 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.322130    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ed81bb\"/\"default-dockercfg-lqlll\""
Sep 09 03:34:40.325433 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.325406    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"]
Sep 09 03:34:40.332973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.332945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trq7w\" (UniqueName: \"kubernetes.io/projected/85b280a2-e283-4d00-bd8f-1fe9c4428a06-kube-api-access-trq7w\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.333124 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.332997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-home\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.333176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.333149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tls-certs\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.333224 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.333190    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tmp-dir\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.333263 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.333249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-dshm\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.333319 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.333302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-model-cache\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.360645 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.360605    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:40.433908 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.433866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trq7w\" (UniqueName: \"kubernetes.io/projected/85b280a2-e283-4d00-bd8f-1fe9c4428a06-kube-api-access-trq7w\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.434126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.433925    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-home\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.434126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.434004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tls-certs\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.434126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.434033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tmp-dir\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.434126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.434067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-dshm\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.434126 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.434107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-model-cache\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.434442 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.434415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-home\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.434510 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.434425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tmp-dir\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.434510 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.434475    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-model-cache\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.436797 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.436768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-dshm\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.437015 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.436993    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tls-certs\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.442452 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.442424    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trq7w\" (UniqueName: \"kubernetes.io/projected/85b280a2-e283-4d00-bd8f-1fe9c4428a06-kube-api-access-trq7w\") pod \"ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") " pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.631149 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.631049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:40.908621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:40.908596    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"]
Sep 09 03:34:40.910354 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:34:40.910322    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85b280a2_e283_4d00_bd8f_1fe9c4428a06.slice/crio-f4777b2cb1e31e96a0dee7d69cfb13a47dab562aba2e6bbb14e921df758b97c1 WatchSource:0}: Error finding container f4777b2cb1e31e96a0dee7d69cfb13a47dab562aba2e6bbb14e921df758b97c1: Status 404 returned error can't find the container with id f4777b2cb1e31e96a0dee7d69cfb13a47dab562aba2e6bbb14e921df758b97c1
Sep 09 03:34:41.048004 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:41.047965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq" event={"ID":"85b280a2-e283-4d00-bd8f-1fe9c4428a06","Type":"ContainerStarted","Data":"373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a"}
Sep 09 03:34:41.048188 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:41.048019    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq" event={"ID":"85b280a2-e283-4d00-bd8f-1fe9c4428a06","Type":"ContainerStarted","Data":"f4777b2cb1e31e96a0dee7d69cfb13a47dab562aba2e6bbb14e921df758b97c1"}
Sep 09 03:34:41.067951 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:41.067893    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq" podStartSLOduration=1.06787111 podStartE2EDuration="1.06787111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:34:41.065142808 +0000 UTC m=+1729.627469351" watchObservedRunningTime="2026-09-09 03:34:41.06787111 +0000 UTC m=+1729.630197655"
Sep 09 03:34:50.631873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:50.631834    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:50.632326 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:50.631886    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:50.644965 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:50.644932    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:34:51.131473 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:34:51.131438    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:35:07.008468 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:07.008420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:07.083265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:07.083221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:09.791908 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:09.791875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:09.879712 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:09.879678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:10.939287 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:10.939254    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"]
Sep 09 03:35:10.942529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:10.942504    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:10.944760 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:10.944732    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-7e9fa8-serving-cert\""
Sep 09 03:35:10.956959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:10.956931    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"]
Sep 09 03:35:10.977766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:10.977730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:11.051866 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.051826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7087231c-4122-4677-81e7-3aa4ba43f856-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.052052 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.051979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.052052 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.052025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.052143 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.052075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcsx8\" (UniqueName: \"kubernetes.io/projected/7087231c-4122-4677-81e7-3aa4ba43f856-kube-api-access-tcsx8\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.054693 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.054663    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:11.152844 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.152807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.153065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.152866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.153065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.152902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcsx8\" (UniqueName: \"kubernetes.io/projected/7087231c-4122-4677-81e7-3aa4ba43f856-kube-api-access-tcsx8\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.153065 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:35:11.153040    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-serving-cert: secret "maas-api-e2e-ipp-cleanup-7e9fa8-serving-cert" not found
Sep 09 03:35:11.153236 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.153076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7087231c-4122-4677-81e7-3aa4ba43f856-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.153236 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:35:11.153147    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls podName:7087231c-4122-4677-81e7-3aa4ba43f856 nodeName:}" failed. No retries permitted until 2026-09-09 03:35:11.653107907 +0000 UTC m=+1760.215434434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h" (UID: "7087231c-4122-4677-81e7-3aa4ba43f856") : secret "maas-api-e2e-ipp-cleanup-7e9fa8-serving-cert" not found
Sep 09 03:35:11.153827 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.153791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7087231c-4122-4677-81e7-3aa4ba43f856-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.155586 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.155565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.171214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.171187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcsx8\" (UniqueName: \"kubernetes.io/projected/7087231c-4122-4677-81e7-3aa4ba43f856-kube-api-access-tcsx8\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.327821 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.327771    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:11.616111 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.616007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:11.658331 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.658300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.661013 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.660985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:11.716176 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.716137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:11.855047 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:11.855008    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:12.130945 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:12.130914    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"]
Sep 09 03:35:12.132138 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:35:12.132105    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7087231c_4122_4677_81e7_3aa4ba43f856.slice/crio-2bdcfc689bb9b5a70b8c48afa59b0ab47f0860d1bd5aae4de0ecc9092e951489 WatchSource:0}: Error finding container 2bdcfc689bb9b5a70b8c48afa59b0ab47f0860d1bd5aae4de0ecc9092e951489: Status 404 returned error can't find the container with id 2bdcfc689bb9b5a70b8c48afa59b0ab47f0860d1bd5aae4de0ecc9092e951489
Sep 09 03:35:12.249479 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:12.249439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h" event={"ID":"7087231c-4122-4677-81e7-3aa4ba43f856","Type":"ContainerStarted","Data":"2bdcfc689bb9b5a70b8c48afa59b0ab47f0860d1bd5aae4de0ecc9092e951489"}
Sep 09 03:35:13.258503 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:13.258459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h" event={"ID":"7087231c-4122-4677-81e7-3aa4ba43f856","Type":"ContainerStarted","Data":"fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9"}
Sep 09 03:35:13.258967 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:13.258583    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:13.276742 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:13.276698    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h" podStartSLOduration=2.922776324 podStartE2EDuration="3.276681403s" totalImagesPullingTime="353.905079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:13.274278897 +0000 UTC m=+1761.836605442" watchObservedRunningTime="2026-09-09 03:35:13.276681403 +0000 UTC m=+1761.839007945"
Sep 09 03:35:19.271070 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:19.271040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:24.336070 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:24.336029    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:34.927225 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:34.927187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:41.705534 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:41.705495    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"]
Sep 09 03:35:41.706122 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:41.705756    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h" podUID="7087231c-4122-4677-81e7-3aa4ba43f856" containerName="maas-api" containerID="cri-o://fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9" gracePeriod=30
Sep 09 03:35:41.735093 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:41.735038    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:41.836522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:41.836484    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:41.937357 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:41.937281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:41.964520 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:41.964454    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:42.056276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.056241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7087231c-4122-4677-81e7-3aa4ba43f856-openshift-service-ca\") pod \"7087231c-4122-4677-81e7-3aa4ba43f856\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") "
Sep 09 03:35:42.056483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.056305    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-metrics-certs\") pod \"7087231c-4122-4677-81e7-3aa4ba43f856\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") "
Sep 09 03:35:42.056483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.056387    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls\") pod \"7087231c-4122-4677-81e7-3aa4ba43f856\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") "
Sep 09 03:35:42.056483 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.056469    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcsx8\" (UniqueName: \"kubernetes.io/projected/7087231c-4122-4677-81e7-3aa4ba43f856-kube-api-access-tcsx8\") pod \"7087231c-4122-4677-81e7-3aa4ba43f856\" (UID: \"7087231c-4122-4677-81e7-3aa4ba43f856\") "
Sep 09 03:35:42.056729 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.056699    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7087231c-4122-4677-81e7-3aa4ba43f856-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7087231c-4122-4677-81e7-3aa4ba43f856" (UID: "7087231c-4122-4677-81e7-3aa4ba43f856"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:35:42.057774 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.056930    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7087231c-4122-4677-81e7-3aa4ba43f856-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:35:42.058939 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.058913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7087231c-4122-4677-81e7-3aa4ba43f856" (UID: "7087231c-4122-4677-81e7-3aa4ba43f856"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:35:42.059181 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.059159    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7087231c-4122-4677-81e7-3aa4ba43f856-kube-api-access-tcsx8" (OuterVolumeSpecName: "kube-api-access-tcsx8") pod "7087231c-4122-4677-81e7-3aa4ba43f856" (UID: "7087231c-4122-4677-81e7-3aa4ba43f856"). InnerVolumeSpecName "kube-api-access-tcsx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:35:42.059632 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.059606    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7087231c-4122-4677-81e7-3aa4ba43f856" (UID: "7087231c-4122-4677-81e7-3aa4ba43f856"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:35:42.117375 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.104892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:42.158350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.158305    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcsx8\" (UniqueName: \"kubernetes.io/projected/7087231c-4122-4677-81e7-3aa4ba43f856-kube-api-access-tcsx8\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:35:42.158350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.158341    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:35:42.158350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.158350    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7087231c-4122-4677-81e7-3aa4ba43f856-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:35:42.320927 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.320882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:42.445519 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.445477    2562 generic.go:356] "Generic (PLEG): container finished" podID="7087231c-4122-4677-81e7-3aa4ba43f856" containerID="fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9" exitCode=0
Sep 09 03:35:42.445750 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.445571    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"
Sep 09 03:35:42.445750 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.445567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h" event={"ID":"7087231c-4122-4677-81e7-3aa4ba43f856","Type":"ContainerDied","Data":"fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9"}
Sep 09 03:35:42.445750 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.445672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h" event={"ID":"7087231c-4122-4677-81e7-3aa4ba43f856","Type":"ContainerDied","Data":"2bdcfc689bb9b5a70b8c48afa59b0ab47f0860d1bd5aae4de0ecc9092e951489"}
Sep 09 03:35:42.445750 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.445692    2562 scope.go:117] "RemoveContainer" containerID="fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9"
Sep 09 03:35:42.458707 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.458687    2562 scope.go:117] "RemoveContainer" containerID="fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9"
Sep 09 03:35:42.459048 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:35:42.459024    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9\": container with ID starting with fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9 not found: ID does not exist" containerID="fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9"
Sep 09 03:35:42.459164 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.459052    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9"} err="failed to get container status \"fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9\": rpc error: code = NotFound desc = could not find container \"fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9\": container with ID starting with fe14537afc072a06e1dff174b9f9fe709212962aaaaa278e4b993a4d0ddddfd9 not found: ID does not exist"
Sep 09 03:35:42.470132 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.470103    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"]
Sep 09 03:35:42.473818 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:42.473782    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7e9fa8-55d6cb7d7-2nk7h"]
Sep 09 03:35:43.939009 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:43.938975    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7087231c-4122-4677-81e7-3aa4ba43f856" path="/var/lib/kubelet/pods/7087231c-4122-4677-81e7-3aa4ba43f856/volumes"
Sep 09 03:35:48.322294 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:48.322255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:48.450410 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:48.450375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:48.616078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:48.615988    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:52.467521 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:35:52.467482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:07.916970 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:07.916930    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:07.921755 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:07.921715    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"]
Sep 09 03:36:07.922044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:07.922016    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6" podUID="f12040f9-2f1d-4b88-8897-3c0c96979bac" containerName="maas-api" containerID="cri-o://683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5" gracePeriod=30
Sep 09 03:36:08.048211 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.048168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:08.191222 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.191187    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:36:08.214146 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.214112    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:08.327957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.327923    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12040f9-2f1d-4b88-8897-3c0c96979bac-openshift-service-ca\") pod \"f12040f9-2f1d-4b88-8897-3c0c96979bac\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") "
Sep 09 03:36:08.327957 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.327960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-maas-api-tls\") pod \"f12040f9-2f1d-4b88-8897-3c0c96979bac\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") "
Sep 09 03:36:08.328212 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.327983    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-metrics-certs\") pod \"f12040f9-2f1d-4b88-8897-3c0c96979bac\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") "
Sep 09 03:36:08.328212 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.328126    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mm2j\" (UniqueName: \"kubernetes.io/projected/f12040f9-2f1d-4b88-8897-3c0c96979bac-kube-api-access-2mm2j\") pod \"f12040f9-2f1d-4b88-8897-3c0c96979bac\" (UID: \"f12040f9-2f1d-4b88-8897-3c0c96979bac\") "
Sep 09 03:36:08.328397 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.328360    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12040f9-2f1d-4b88-8897-3c0c96979bac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f12040f9-2f1d-4b88-8897-3c0c96979bac" (UID: "f12040f9-2f1d-4b88-8897-3c0c96979bac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:36:08.328558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.328495    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12040f9-2f1d-4b88-8897-3c0c96979bac-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:08.330334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.330305    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f12040f9-2f1d-4b88-8897-3c0c96979bac" (UID: "f12040f9-2f1d-4b88-8897-3c0c96979bac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:36:08.330796 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.330772    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f12040f9-2f1d-4b88-8897-3c0c96979bac" (UID: "f12040f9-2f1d-4b88-8897-3c0c96979bac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:36:08.330852 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.330792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12040f9-2f1d-4b88-8897-3c0c96979bac-kube-api-access-2mm2j" (OuterVolumeSpecName: "kube-api-access-2mm2j") pod "f12040f9-2f1d-4b88-8897-3c0c96979bac" (UID: "f12040f9-2f1d-4b88-8897-3c0c96979bac"). InnerVolumeSpecName "kube-api-access-2mm2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:36:08.429459 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.429359    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:08.429459 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.429399    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mm2j\" (UniqueName: \"kubernetes.io/projected/f12040f9-2f1d-4b88-8897-3c0c96979bac-kube-api-access-2mm2j\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:08.429459 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.429410    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12040f9-2f1d-4b88-8897-3c0c96979bac-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:08.438746 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.438705    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:08.546476 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.546436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:08.616789 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.616753    2562 generic.go:356] "Generic (PLEG): container finished" podID="f12040f9-2f1d-4b88-8897-3c0c96979bac" containerID="683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5" exitCode=0
Sep 09 03:36:08.616959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.616801    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"
Sep 09 03:36:08.616959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.616807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6" event={"ID":"f12040f9-2f1d-4b88-8897-3c0c96979bac","Type":"ContainerDied","Data":"683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5"}
Sep 09 03:36:08.616959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.616865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6" event={"ID":"f12040f9-2f1d-4b88-8897-3c0c96979bac","Type":"ContainerDied","Data":"cb7658d419da8f2617bdd1858e1681fda0da8a9c873374cb256260283b09c0ad"}
Sep 09 03:36:08.616959 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.616890    2562 scope.go:117] "RemoveContainer" containerID="683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5"
Sep 09 03:36:08.628271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.628244    2562 scope.go:117] "RemoveContainer" containerID="683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5"
Sep 09 03:36:08.628624 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:36:08.628606    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5\": container with ID starting with 683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5 not found: ID does not exist" containerID="683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5"
Sep 09 03:36:08.628684 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.628645    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5"} err="failed to get container status \"683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5\": rpc error: code = NotFound desc = could not find container \"683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5\": container with ID starting with 683250338b5285626b52c60cb2f76a0397bdddae7f2cfb29e98392b291d485e5 not found: ID does not exist"
Sep 09 03:36:08.641276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.641245    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"]
Sep 09 03:36:08.652825 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.652788    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed81bb-57cffbd7dd-g9cq6"]
Sep 09 03:36:08.948458 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:08.948414    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:09.940179 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:09.940141    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12040f9-2f1d-4b88-8897-3c0c96979bac" path="/var/lib/kubelet/pods/f12040f9-2f1d-4b88-8897-3c0c96979bac/volumes"
Sep 09 03:36:13.868443 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:13.868397    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"]
Sep 09 03:36:13.952044 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:13.952004    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:21.954906 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:21.954844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:22.527230 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:22.527191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:22.587205 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:22.587167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:28.890053 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:28.890001    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq" podUID="85b280a2-e283-4d00-bd8f-1fe9c4428a06" containerName="main" containerID="cri-o://373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a" gracePeriod=45
Sep 09 03:36:29.150021 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.149943    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:36:29.245823 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.245779    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trq7w\" (UniqueName: \"kubernetes.io/projected/85b280a2-e283-4d00-bd8f-1fe9c4428a06-kube-api-access-trq7w\") pod \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") "
Sep 09 03:36:29.246036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.245843    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tls-certs\") pod \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") "
Sep 09 03:36:29.246036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.245866    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-model-cache\") pod \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") "
Sep 09 03:36:29.246036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.245927    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-dshm\") pod \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") "
Sep 09 03:36:29.246036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.245965    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-home\") pod \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") "
Sep 09 03:36:29.246036 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.245987    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tmp-dir\") pod \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\" (UID: \"85b280a2-e283-4d00-bd8f-1fe9c4428a06\") "
Sep 09 03:36:29.246332 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.246249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-home" (OuterVolumeSpecName: "home") pod "85b280a2-e283-4d00-bd8f-1fe9c4428a06" (UID: "85b280a2-e283-4d00-bd8f-1fe9c4428a06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:36:29.246332 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.246261    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-model-cache" (OuterVolumeSpecName: "model-cache") pod "85b280a2-e283-4d00-bd8f-1fe9c4428a06" (UID: "85b280a2-e283-4d00-bd8f-1fe9c4428a06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:36:29.246440 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.246341    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85b280a2-e283-4d00-bd8f-1fe9c4428a06" (UID: "85b280a2-e283-4d00-bd8f-1fe9c4428a06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:36:29.248385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.248356    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-dshm" (OuterVolumeSpecName: "dshm") pod "85b280a2-e283-4d00-bd8f-1fe9c4428a06" (UID: "85b280a2-e283-4d00-bd8f-1fe9c4428a06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:36:29.248385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.248365    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85b280a2-e283-4d00-bd8f-1fe9c4428a06" (UID: "85b280a2-e283-4d00-bd8f-1fe9c4428a06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:36:29.248385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.248370    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b280a2-e283-4d00-bd8f-1fe9c4428a06-kube-api-access-trq7w" (OuterVolumeSpecName: "kube-api-access-trq7w") pod "85b280a2-e283-4d00-bd8f-1fe9c4428a06" (UID: "85b280a2-e283-4d00-bd8f-1fe9c4428a06"). InnerVolumeSpecName "kube-api-access-trq7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:36:29.347272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.347224    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:29.347272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.347263    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trq7w\" (UniqueName: \"kubernetes.io/projected/85b280a2-e283-4d00-bd8f-1fe9c4428a06-kube-api-access-trq7w\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:29.347272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.347275    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b280a2-e283-4d00-bd8f-1fe9c4428a06-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:29.347272 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.347284    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:29.347619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.347293    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:29.347619 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.347300    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b280a2-e283-4d00-bd8f-1fe9c4428a06-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:36:29.751209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.751168    2562 generic.go:356] "Generic (PLEG): container finished" podID="85b280a2-e283-4d00-bd8f-1fe9c4428a06" containerID="373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a" exitCode=0
Sep 09 03:36:29.751209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.751212    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"
Sep 09 03:36:29.751437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.751217    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq" event={"ID":"85b280a2-e283-4d00-bd8f-1fe9c4428a06","Type":"ContainerDied","Data":"373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a"}
Sep 09 03:36:29.751437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.751327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq" event={"ID":"85b280a2-e283-4d00-bd8f-1fe9c4428a06","Type":"ContainerDied","Data":"f4777b2cb1e31e96a0dee7d69cfb13a47dab562aba2e6bbb14e921df758b97c1"}
Sep 09 03:36:29.751437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.751348    2562 scope.go:117] "RemoveContainer" containerID="373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a"
Sep 09 03:36:29.761934 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.761912    2562 scope.go:117] "RemoveContainer" containerID="373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a"
Sep 09 03:36:29.762343 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:36:29.762306    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a\": container with ID starting with 373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a not found: ID does not exist" containerID="373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a"
Sep 09 03:36:29.762343 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.762340    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a"} err="failed to get container status \"373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a\": rpc error: code = NotFound desc = could not find container \"373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a\": container with ID starting with 373a036611a59f8b621d161e3adb29b22144229092180c25efeac6dd16530d4a not found: ID does not exist"
Sep 09 03:36:29.773662 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.773614    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"]
Sep 09 03:36:29.777935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.777892    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ed81bb/ipp-route-ed81bb-kserve-5b7c4cdfff-m9tgq"]
Sep 09 03:36:29.946690 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:29.946645    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b280a2-e283-4d00-bd8f-1fe9c4428a06" path="/var/lib/kubelet/pods/85b280a2-e283-4d00-bd8f-1fe9c4428a06/volumes"
Sep 09 03:36:50.164761 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:50.164710    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:51.103067 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:51.103017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:51.687676 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:51.687638    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:52.297091 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:52.297057    2562 scope.go:117] "RemoveContainer" containerID="ccb5fcc9e9a98b354bcf33adf868b2f8f95d5403928028031279370c4b916eea"
Sep 09 03:36:56.748824 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:56.748786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:56.839334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:56.839300    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:57.054076 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:57.054032    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:57.624903 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:36:57.624863    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:08.123863 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.123827    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"]
Sep 09 03:37:08.124432 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.124054    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz" podUID="90657ced-fd95-4bef-ba46-879d9fa682ae" containerName="maas-api" containerID="cri-o://3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f" gracePeriod=30
Sep 09 03:37:08.162265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.162217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:08.271279 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.271196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:08.390990 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.390917    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:37:08.410223 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.410187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:08.439059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.439029    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90657ced-fd95-4bef-ba46-879d9fa682ae-openshift-service-ca\") pod \"90657ced-fd95-4bef-ba46-879d9fa682ae\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") "
Sep 09 03:37:08.439239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.439110    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-metrics-certs\") pod \"90657ced-fd95-4bef-ba46-879d9fa682ae\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") "
Sep 09 03:37:08.439239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.439181    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls\") pod \"90657ced-fd95-4bef-ba46-879d9fa682ae\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") "
Sep 09 03:37:08.439376 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.439303    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84jkc\" (UniqueName: \"kubernetes.io/projected/90657ced-fd95-4bef-ba46-879d9fa682ae-kube-api-access-84jkc\") pod \"90657ced-fd95-4bef-ba46-879d9fa682ae\" (UID: \"90657ced-fd95-4bef-ba46-879d9fa682ae\") "
Sep 09 03:37:08.439376 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.439351    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90657ced-fd95-4bef-ba46-879d9fa682ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90657ced-fd95-4bef-ba46-879d9fa682ae" (UID: "90657ced-fd95-4bef-ba46-879d9fa682ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:08.439706 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.439677    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90657ced-fd95-4bef-ba46-879d9fa682ae-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:08.441447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.441417    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90657ced-fd95-4bef-ba46-879d9fa682ae" (UID: "90657ced-fd95-4bef-ba46-879d9fa682ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:08.441558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.441440    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90657ced-fd95-4bef-ba46-879d9fa682ae" (UID: "90657ced-fd95-4bef-ba46-879d9fa682ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:08.441558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.441519    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90657ced-fd95-4bef-ba46-879d9fa682ae-kube-api-access-84jkc" (OuterVolumeSpecName: "kube-api-access-84jkc") pod "90657ced-fd95-4bef-ba46-879d9fa682ae" (UID: "90657ced-fd95-4bef-ba46-879d9fa682ae"). InnerVolumeSpecName "kube-api-access-84jkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:08.491047 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.491005    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:08.540892 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.540846    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:08.540892 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.540885    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84jkc\" (UniqueName: \"kubernetes.io/projected/90657ced-fd95-4bef-ba46-879d9fa682ae-kube-api-access-84jkc\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:08.540892 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.540899    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90657ced-fd95-4bef-ba46-879d9fa682ae-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:08.771659 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.771559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:08.992003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.991967    2562 generic.go:356] "Generic (PLEG): container finished" podID="90657ced-fd95-4bef-ba46-879d9fa682ae" containerID="3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f" exitCode=0
Sep 09 03:37:08.992003 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.992004    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz" event={"ID":"90657ced-fd95-4bef-ba46-879d9fa682ae","Type":"ContainerDied","Data":"3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f"}
Sep 09 03:37:08.992265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.992024    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"
Sep 09 03:37:08.992265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.992046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz" event={"ID":"90657ced-fd95-4bef-ba46-879d9fa682ae","Type":"ContainerDied","Data":"b9d275c3938be5edf5405b41d781ab62acf1e1c5f8d3ee2e4e660088710db6c3"}
Sep 09 03:37:08.992265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:08.992068    2562 scope.go:117] "RemoveContainer" containerID="3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f"
Sep 09 03:37:09.005494 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:09.005465    2562 scope.go:117] "RemoveContainer" containerID="3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f"
Sep 09 03:37:09.006099 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:37:09.006075    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f\": container with ID starting with 3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f not found: ID does not exist" containerID="3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f"
Sep 09 03:37:09.006209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:09.006106    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f"} err="failed to get container status \"3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f\": rpc error: code = NotFound desc = could not find container \"3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f\": container with ID starting with 3aca275f4acb2563b9f17c88792f96d8ebbd15f43d7111f0589f29d29c8b156f not found: ID does not exist"
Sep 09 03:37:09.016949 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:09.016922    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"]
Sep 09 03:37:09.019518 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:09.019495    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aacd65-6bd4df9cf-8rzsz"]
Sep 09 03:37:09.938675 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:09.938641    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90657ced-fd95-4bef-ba46-879d9fa682ae" path="/var/lib/kubelet/pods/90657ced-fd95-4bef-ba46-879d9fa682ae/volumes"
Sep 09 03:37:14.650319 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:14.650218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:14.751214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:14.751173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:14.892406 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:14.892370    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"]
Sep 09 03:37:14.892664 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:14.892635    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy" containerID="cri-o://ed02db8694c12662e52a0d994c9d951ae2158da511a92e466f8869c942a0d078" gracePeriod=30
Sep 09 03:37:14.921265 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:14.921168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:15.360134 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:15.360091    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:20.063859 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.063817    2562 generic.go:356] "Generic (PLEG): container finished" podID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerID="ed02db8694c12662e52a0d994c9d951ae2158da511a92e466f8869c942a0d078" exitCode=0
Sep 09 03:37:20.064256 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.063881    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" event={"ID":"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d","Type":"ContainerDied","Data":"ed02db8694c12662e52a0d994c9d951ae2158da511a92e466f8869c942a0d078"}
Sep 09 03:37:20.064256 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.063950    2562 scope.go:117] "RemoveContainer" containerID="c4187379f8fc6b31843758eb056a4f66bc219d95e6868c47877e8c786870ddb5"
Sep 09 03:37:20.170669 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.170589    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:37:20.255482 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255436    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-socket\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.255698 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255499    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istiod-ca-cert\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.255698 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255613    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-podinfo\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.255781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-credential-socket\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.255832 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255792    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-token\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.255832 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255816    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:20.255935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-data\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.255935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255869    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-certs\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.255935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255904    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-envoy\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.256099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255975    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nqhz\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-kube-api-access-5nqhz\") pod \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\" (UID: \"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d\") "
Sep 09 03:37:20.256099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.255995    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:20.256207 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.256018    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:20.256207 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.256103    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:20.256207 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.256134    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:20.256563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.256416    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:20.256563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.256528    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:20.256563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.256574    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:20.256563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.256588    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:20.256563 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.256601    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:20.258032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.258009    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:37:20.258254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.258222    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:20.258254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.258238    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-kube-api-access-5nqhz" (OuterVolumeSpecName: "kube-api-access-5nqhz") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "kube-api-access-5nqhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:20.258437 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.258284    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" (UID: "d3bdddde-bd1c-4033-a7b2-fca98c7ae17d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:20.358027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.357987    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nqhz\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-kube-api-access-5nqhz\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:20.358027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.358019    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:20.358027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.358030    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:20.358027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:20.358039    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:21.071692 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:21.071658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5" event={"ID":"d3bdddde-bd1c-4033-a7b2-fca98c7ae17d","Type":"ContainerDied","Data":"3060adb194bc9b8b500ef4d7c35cb54b426ff17d82ab53d04dda4cc7f38468e5"}
Sep 09 03:37:21.071692 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:21.071693    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"
Sep 09 03:37:21.072174 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:21.071721    2562 scope.go:117] "RemoveContainer" containerID="ed02db8694c12662e52a0d994c9d951ae2158da511a92e466f8869c942a0d078"
Sep 09 03:37:21.094715 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:21.094679    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"]
Sep 09 03:37:21.101336 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:21.101304    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-aacd65-openshift-default-5d5fb7d67b-r92w5"]
Sep 09 03:37:21.938506 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:21.938468    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" path="/var/lib/kubelet/pods/d3bdddde-bd1c-4033-a7b2-fca98c7ae17d/volumes"
Sep 09 03:37:25.716298 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:25.716258    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"]
Sep 09 03:37:25.716864 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:25.716599    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k" podUID="65a8175d-04dc-45a2-84d6-224f1aa65f95" containerName="maas-api" containerID="cri-o://23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411" gracePeriod=30
Sep 09 03:37:25.753453 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:25.753410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:25.822522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:25.822486    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:25.900078 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:25.900017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:25.983186 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:25.983116    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:37:25.995399 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:25.995367    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:26.071749 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.071717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:26.105426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.105391    2562 generic.go:356] "Generic (PLEG): container finished" podID="65a8175d-04dc-45a2-84d6-224f1aa65f95" containerID="23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411" exitCode=0
Sep 09 03:37:26.105629 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.105477    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k" event={"ID":"65a8175d-04dc-45a2-84d6-224f1aa65f95","Type":"ContainerDied","Data":"23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411"}
Sep 09 03:37:26.105629 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.105494    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"
Sep 09 03:37:26.105629 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.105574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k" event={"ID":"65a8175d-04dc-45a2-84d6-224f1aa65f95","Type":"ContainerDied","Data":"8c0e357c765b05aee2b1a45b96baf62cb5fe3e22215d74c0e165e2d87e0aea1c"}
Sep 09 03:37:26.105629 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.105583    2562 scope.go:117] "RemoveContainer" containerID="23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411"
Sep 09 03:37:26.116616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.116589    2562 scope.go:117] "RemoveContainer" containerID="23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411"
Sep 09 03:37:26.116964 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:37:26.116943    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411\": container with ID starting with 23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411 not found: ID does not exist" containerID="23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411"
Sep 09 03:37:26.117039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.116975    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411"} err="failed to get container status \"23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411\": rpc error: code = NotFound desc = could not find container \"23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411\": container with ID starting with 23786b2388afa394ba4f668b4b50577db11a8b677ff8b4bc58a39234e3060411 not found: ID does not exist"
Sep 09 03:37:26.117039 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.117002    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a8175d-04dc-45a2-84d6-224f1aa65f95-openshift-service-ca\") pod \"65a8175d-04dc-45a2-84d6-224f1aa65f95\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") "
Sep 09 03:37:26.117127 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.117052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-maas-api-tls\") pod \"65a8175d-04dc-45a2-84d6-224f1aa65f95\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") "
Sep 09 03:37:26.117167 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.117125    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-metrics-certs\") pod \"65a8175d-04dc-45a2-84d6-224f1aa65f95\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") "
Sep 09 03:37:26.117214 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.117181    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfsk7\" (UniqueName: \"kubernetes.io/projected/65a8175d-04dc-45a2-84d6-224f1aa65f95-kube-api-access-zfsk7\") pod \"65a8175d-04dc-45a2-84d6-224f1aa65f95\" (UID: \"65a8175d-04dc-45a2-84d6-224f1aa65f95\") "
Sep 09 03:37:26.117347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.117325    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a8175d-04dc-45a2-84d6-224f1aa65f95-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65a8175d-04dc-45a2-84d6-224f1aa65f95" (UID: "65a8175d-04dc-45a2-84d6-224f1aa65f95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:26.118008 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.117981    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a8175d-04dc-45a2-84d6-224f1aa65f95-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:26.119595 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.119569    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65a8175d-04dc-45a2-84d6-224f1aa65f95" (UID: "65a8175d-04dc-45a2-84d6-224f1aa65f95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:26.119692 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.119646    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a8175d-04dc-45a2-84d6-224f1aa65f95-kube-api-access-zfsk7" (OuterVolumeSpecName: "kube-api-access-zfsk7") pod "65a8175d-04dc-45a2-84d6-224f1aa65f95" (UID: "65a8175d-04dc-45a2-84d6-224f1aa65f95"). InnerVolumeSpecName "kube-api-access-zfsk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:26.119741 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.119719    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65a8175d-04dc-45a2-84d6-224f1aa65f95" (UID: "65a8175d-04dc-45a2-84d6-224f1aa65f95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:26.218837 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.218797    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:26.218837 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.218831    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a8175d-04dc-45a2-84d6-224f1aa65f95-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:26.219077 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.218865    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsk7\" (UniqueName: \"kubernetes.io/projected/65a8175d-04dc-45a2-84d6-224f1aa65f95-kube-api-access-zfsk7\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:26.439430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.439388    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"]
Sep 09 03:37:26.449640 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.449595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:26.450205 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:26.450182    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1a85fa-655fb65f87-ch68k"]
Sep 09 03:37:27.939364 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:27.939331    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a8175d-04dc-45a2-84d6-224f1aa65f95" path="/var/lib/kubelet/pods/65a8175d-04dc-45a2-84d6-224f1aa65f95/volumes"
Sep 09 03:37:32.389236 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:32.389179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:32.489854 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:32.489815    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:32.633724 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:32.633690    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"]
Sep 09 03:37:32.633956 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:32.633921    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy" containerID="cri-o://f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e" gracePeriod=30
Sep 09 03:37:32.651065 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:32.650946    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:33.127693 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:33.127653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:35.622352 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:35.622298    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:37:35.622927 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:35.622368    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:37:37.899325 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:37.899295    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:37:38.037147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037112    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-podinfo\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037147 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037153    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-envoy\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037172    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-socket\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037190    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-credential-socket\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037210    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istiod-ca-cert\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037237    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8ptx\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-kube-api-access-v8ptx\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037275    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-token\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-certs\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037419 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037330    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-data\") pod \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\" (UID: \"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5\") "
Sep 09 03:37:38.037815 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037663    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:38.037815 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:38.037925 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037846    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:38.037925 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.037860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:38.038087 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.038061    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:38.039795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.039770    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-kube-api-access-v8ptx" (OuterVolumeSpecName: "kube-api-access-v8ptx") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "kube-api-access-v8ptx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:38.039864 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.039793    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:38.040005 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.039982    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:37:38.040098 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.040077    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" (UID: "b81a990f-b8d2-43b2-8dbe-ee71d0f213b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:38.139023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.138981    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.139023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.139016    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.139023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.139025    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.139023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.139033    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.139299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.139043    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.139299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.139052    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ptx\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-kube-api-access-v8ptx\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.139299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.139059    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.139299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.139068    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.139299 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.139075    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:38.179695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.179660    2562 generic.go:356] "Generic (PLEG): container finished" podID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerID="f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e" exitCode=0
Sep 09 03:37:38.179870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.179735    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"
Sep 09 03:37:38.179870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.179739    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" event={"ID":"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5","Type":"ContainerDied","Data":"f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e"}
Sep 09 03:37:38.179870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.179791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs" event={"ID":"b81a990f-b8d2-43b2-8dbe-ee71d0f213b5","Type":"ContainerDied","Data":"3d2fec05cccc1c993ceb2bc7b451d75cbb2b4c5ef0e7d94b23f23ac31d9a29ec"}
Sep 09 03:37:38.179870 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.179813    2562 scope.go:117] "RemoveContainer" containerID="f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e"
Sep 09 03:37:38.190193 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.190173    2562 scope.go:117] "RemoveContainer" containerID="afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650"
Sep 09 03:37:38.199275 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.199253    2562 scope.go:117] "RemoveContainer" containerID="f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e"
Sep 09 03:37:38.199623 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:37:38.199602    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e\": container with ID starting with f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e not found: ID does not exist" containerID="f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e"
Sep 09 03:37:38.199701 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.199630    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e"} err="failed to get container status \"f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e\": rpc error: code = NotFound desc = could not find container \"f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e\": container with ID starting with f28a49074eaac0a3049b5f907abbd8db9d78b49291d1764261946d7ca10a345e not found: ID does not exist"
Sep 09 03:37:38.199701 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.199648    2562 scope.go:117] "RemoveContainer" containerID="afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650"
Sep 09 03:37:38.199925 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:37:38.199906    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650\": container with ID starting with afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650 not found: ID does not exist" containerID="afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650"
Sep 09 03:37:38.199964 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.199933    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650"} err="failed to get container status \"afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650\": rpc error: code = NotFound desc = could not find container \"afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650\": container with ID starting with afdb7b31c147f9f81863f94b73e32bf7fc8a5827ef66dde2967259b2f7ac9650 not found: ID does not exist"
Sep 09 03:37:38.205133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.205104    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"]
Sep 09 03:37:38.212981 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:38.212947    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1a85fa-openshift-default-5d795697d4-ktvgs"]
Sep 09 03:37:39.938522 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:39.938489    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" path="/var/lib/kubelet/pods/b81a990f-b8d2-43b2-8dbe-ee71d0f213b5/volumes"
Sep 09 03:37:42.502237 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:42.502191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:43.564865 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.564814    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"]
Sep 09 03:37:43.565696 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.565636    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz" podUID="38a18f96-7589-4adc-86bc-fb6ae34041e4" containerName="maas-api" containerID="cri-o://baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc" gracePeriod=30
Sep 09 03:37:43.599250 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.599218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:43.701874 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.701836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:43.803968 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.803938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:43.824274 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.824191    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:37:43.895848 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.895812    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-maas-api-tls\") pod \"38a18f96-7589-4adc-86bc-fb6ae34041e4\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") "
Sep 09 03:37:43.896023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.895866    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a18f96-7589-4adc-86bc-fb6ae34041e4-openshift-service-ca\") pod \"38a18f96-7589-4adc-86bc-fb6ae34041e4\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") "
Sep 09 03:37:43.896023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.895904    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-metrics-certs\") pod \"38a18f96-7589-4adc-86bc-fb6ae34041e4\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") "
Sep 09 03:37:43.896023 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.895962    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm8pl\" (UniqueName: \"kubernetes.io/projected/38a18f96-7589-4adc-86bc-fb6ae34041e4-kube-api-access-rm8pl\") pod \"38a18f96-7589-4adc-86bc-fb6ae34041e4\" (UID: \"38a18f96-7589-4adc-86bc-fb6ae34041e4\") "
Sep 09 03:37:43.896323 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.896295    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a18f96-7589-4adc-86bc-fb6ae34041e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38a18f96-7589-4adc-86bc-fb6ae34041e4" (UID: "38a18f96-7589-4adc-86bc-fb6ae34041e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:43.898444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.898399    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "38a18f96-7589-4adc-86bc-fb6ae34041e4" (UID: "38a18f96-7589-4adc-86bc-fb6ae34041e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:43.898444 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.898425    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a18f96-7589-4adc-86bc-fb6ae34041e4-kube-api-access-rm8pl" (OuterVolumeSpecName: "kube-api-access-rm8pl") pod "38a18f96-7589-4adc-86bc-fb6ae34041e4" (UID: "38a18f96-7589-4adc-86bc-fb6ae34041e4"). InnerVolumeSpecName "kube-api-access-rm8pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:43.898670 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.898463    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38a18f96-7589-4adc-86bc-fb6ae34041e4" (UID: "38a18f96-7589-4adc-86bc-fb6ae34041e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:43.899209 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.899173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:43.997603 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.997534    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm8pl\" (UniqueName: \"kubernetes.io/projected/38a18f96-7589-4adc-86bc-fb6ae34041e4-kube-api-access-rm8pl\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:43.998643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.997595    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:43.998643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.997973    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a18f96-7589-4adc-86bc-fb6ae34041e4-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:43.998643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:43.997988    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a18f96-7589-4adc-86bc-fb6ae34041e4-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:37:44.205766 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.205662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:44.217878 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.217845    2562 generic.go:356] "Generic (PLEG): container finished" podID="38a18f96-7589-4adc-86bc-fb6ae34041e4" containerID="baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc" exitCode=0
Sep 09 03:37:44.218054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.217904    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"
Sep 09 03:37:44.218054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.217928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz" event={"ID":"38a18f96-7589-4adc-86bc-fb6ae34041e4","Type":"ContainerDied","Data":"baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc"}
Sep 09 03:37:44.218054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.218007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz" event={"ID":"38a18f96-7589-4adc-86bc-fb6ae34041e4","Type":"ContainerDied","Data":"ec89f681dc392be5910f78a3cc56a013123d3a7d861d8b83260683b3128ccd76"}
Sep 09 03:37:44.218054 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.218031    2562 scope.go:117] "RemoveContainer" containerID="baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc"
Sep 09 03:37:44.228360 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.228340    2562 scope.go:117] "RemoveContainer" containerID="baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc"
Sep 09 03:37:44.228731 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:37:44.228712    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc\": container with ID starting with baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc not found: ID does not exist" containerID="baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc"
Sep 09 03:37:44.228814 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.228744    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc"} err="failed to get container status \"baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc\": rpc error: code = NotFound desc = could not find container \"baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc\": container with ID starting with baa87b60d8dcdb034565ace093542a68033d27556da85a9d05897c2a8cf073fc not found: ID does not exist"
Sep 09 03:37:44.234143 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.234107    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"]
Sep 09 03:37:44.238185 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:44.238158    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-cd1da6-77bb4cfb6c-qfgbz"]
Sep 09 03:37:45.939966 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:45.939931    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a18f96-7589-4adc-86bc-fb6ae34041e4" path="/var/lib/kubelet/pods/38a18f96-7589-4adc-86bc-fb6ae34041e4/volumes"
Sep 09 03:37:50.393280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:50.393242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:50.495804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:50.495760    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:50.689006 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:50.688915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:51.187103 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:51.187068    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:51.287731 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:51.287693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:52.038851 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:37:52.038812    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:15.134144 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:38:15.134098    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:15.191252 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:38:15.191211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:52.361239 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:38:52.361201    2562 scope.go:117] "RemoveContainer" containerID="f72978d0e1417568ed6454af9283a2ecba120339da242f6385f4e4ada6f2c339"
Sep 09 03:39:16.306308 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:39:16.306265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:17.861835 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:39:17.861787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:36.425271 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:36.425172    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"]
Sep 09 03:40:36.425910 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:36.425458    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q" podUID="2b17c61c-8030-44d4-8ca0-5344cfaafefb" containerName="manager" containerID="cri-o://da5cb2ac05c7dfbabee823cae21eee0974411ff3f7a4f208c933ef6e9d0f762b" gracePeriod=10
Sep 09 03:40:37.218387 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.218349    2562 generic.go:356] "Generic (PLEG): container finished" podID="2b17c61c-8030-44d4-8ca0-5344cfaafefb" containerID="da5cb2ac05c7dfbabee823cae21eee0974411ff3f7a4f208c933ef6e9d0f762b" exitCode=0
Sep 09 03:40:37.218614 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.218408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q" event={"ID":"2b17c61c-8030-44d4-8ca0-5344cfaafefb","Type":"ContainerDied","Data":"da5cb2ac05c7dfbabee823cae21eee0974411ff3f7a4f208c933ef6e9d0f762b"}
Sep 09 03:40:37.287759 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.287731    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:40:37.320402 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.320309    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl9q8\" (UniqueName: \"kubernetes.io/projected/2b17c61c-8030-44d4-8ca0-5344cfaafefb-kube-api-access-vl9q8\") pod \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\" (UID: \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\") "
Sep 09 03:40:37.320606 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.320402    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b17c61c-8030-44d4-8ca0-5344cfaafefb-extensions-socket-volume\") pod \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\" (UID: \"2b17c61c-8030-44d4-8ca0-5344cfaafefb\") "
Sep 09 03:40:37.320867 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.320838    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b17c61c-8030-44d4-8ca0-5344cfaafefb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2b17c61c-8030-44d4-8ca0-5344cfaafefb" (UID: "2b17c61c-8030-44d4-8ca0-5344cfaafefb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:40:37.322820 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.322793    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b17c61c-8030-44d4-8ca0-5344cfaafefb-kube-api-access-vl9q8" (OuterVolumeSpecName: "kube-api-access-vl9q8") pod "2b17c61c-8030-44d4-8ca0-5344cfaafefb" (UID: "2b17c61c-8030-44d4-8ca0-5344cfaafefb"). InnerVolumeSpecName "kube-api-access-vl9q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:40:37.421148 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.421112    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b17c61c-8030-44d4-8ca0-5344cfaafefb-extensions-socket-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:40:37.421148 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:37.421141    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl9q8\" (UniqueName: \"kubernetes.io/projected/2b17c61c-8030-44d4-8ca0-5344cfaafefb-kube-api-access-vl9q8\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:40:38.225713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:38.225626    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"
Sep 09 03:40:38.225713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:38.225649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q" event={"ID":"2b17c61c-8030-44d4-8ca0-5344cfaafefb","Type":"ContainerDied","Data":"14a1d5bed109d4e0135693b8dc23e25b7721166c77391c051722626ab61bfed3"}
Sep 09 03:40:38.225713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:38.225705    2562 scope.go:117] "RemoveContainer" containerID="da5cb2ac05c7dfbabee823cae21eee0974411ff3f7a4f208c933ef6e9d0f762b"
Sep 09 03:40:38.246493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:38.246453    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"]
Sep 09 03:40:38.250783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:38.250753    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6dl7q"]
Sep 09 03:40:39.938059 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:40:39.938024    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b17c61c-8030-44d4-8ca0-5344cfaafefb" path="/var/lib/kubelet/pods/2b17c61c-8030-44d4-8ca0-5344cfaafefb/volumes"
Sep 09 03:41:37.134026 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.133990    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"]
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134422    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134436    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134447    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90657ced-fd95-4bef-ba46-879d9fa682ae" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134454    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90657ced-fd95-4bef-ba46-879d9fa682ae" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134472    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b17c61c-8030-44d4-8ca0-5344cfaafefb" containerName="manager"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134478    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b17c61c-8030-44d4-8ca0-5344cfaafefb" containerName="manager"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134487    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b280a2-e283-4d00-bd8f-1fe9c4428a06" containerName="main"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134493    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b280a2-e283-4d00-bd8f-1fe9c4428a06" containerName="main"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134502    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a8175d-04dc-45a2-84d6-224f1aa65f95" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134507    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a8175d-04dc-45a2-84d6-224f1aa65f95" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134514    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134519    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134524    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134531    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134556    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12040f9-2f1d-4b88-8897-3c0c96979bac" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134562    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12040f9-2f1d-4b88-8897-3c0c96979bac" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134570    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7087231c-4122-4677-81e7-3aa4ba43f856" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134575    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7087231c-4122-4677-81e7-3aa4ba43f856" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134610    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a18f96-7589-4adc-86bc-fb6ae34041e4" containerName="maas-api"
Sep 09 03:41:37.134621 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134620    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a18f96-7589-4adc-86bc-fb6ae34041e4" containerName="maas-api"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134688    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134697    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a8175d-04dc-45a2-84d6-224f1aa65f95" containerName="maas-api"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134706    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134718    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7087231c-4122-4677-81e7-3aa4ba43f856" containerName="maas-api"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134724    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b81a990f-b8d2-43b2-8dbe-ee71d0f213b5" containerName="istio-proxy"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134729    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134735    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b280a2-e283-4d00-bd8f-1fe9c4428a06" containerName="main"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134742    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b17c61c-8030-44d4-8ca0-5344cfaafefb" containerName="manager"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134748    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12040f9-2f1d-4b88-8897-3c0c96979bac" containerName="maas-api"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134753    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a18f96-7589-4adc-86bc-fb6ae34041e4" containerName="maas-api"
Sep 09 03:41:37.135246 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.134759    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="90657ced-fd95-4bef-ba46-879d9fa682ae" containerName="maas-api"
Sep 09 03:41:37.137932 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.137912    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:37.140294 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.140273    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p7lmw\""
Sep 09 03:41:37.150111 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.150083    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"]
Sep 09 03:41:37.188979 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.188935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53de0404-d2b4-4814-8e32-b0f6b6c84ccb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-f5rgs\" (UID: \"53de0404-d2b4-4814-8e32-b0f6b6c84ccb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:37.189158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.189080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74cbt\" (UniqueName: \"kubernetes.io/projected/53de0404-d2b4-4814-8e32-b0f6b6c84ccb-kube-api-access-74cbt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-f5rgs\" (UID: \"53de0404-d2b4-4814-8e32-b0f6b6c84ccb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:37.289774 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.289718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74cbt\" (UniqueName: \"kubernetes.io/projected/53de0404-d2b4-4814-8e32-b0f6b6c84ccb-kube-api-access-74cbt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-f5rgs\" (UID: \"53de0404-d2b4-4814-8e32-b0f6b6c84ccb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:37.289981 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.289826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53de0404-d2b4-4814-8e32-b0f6b6c84ccb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-f5rgs\" (UID: \"53de0404-d2b4-4814-8e32-b0f6b6c84ccb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:37.290184 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.290166    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53de0404-d2b4-4814-8e32-b0f6b6c84ccb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-f5rgs\" (UID: \"53de0404-d2b4-4814-8e32-b0f6b6c84ccb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:37.301040 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.300999    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74cbt\" (UniqueName: \"kubernetes.io/projected/53de0404-d2b4-4814-8e32-b0f6b6c84ccb-kube-api-access-74cbt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-f5rgs\" (UID: \"53de0404-d2b4-4814-8e32-b0f6b6c84ccb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:37.448982 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.448877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:37.731979 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:37.731942    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"]
Sep 09 03:41:37.733667 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:41:37.733631    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53de0404_d2b4_4814_8e32_b0f6b6c84ccb.slice/crio-0f2435925cfea1a0cdf42e7d55551013f8f7e4bc13e525e9021e1f022045e11c WatchSource:0}: Error finding container 0f2435925cfea1a0cdf42e7d55551013f8f7e4bc13e525e9021e1f022045e11c: Status 404 returned error can't find the container with id 0f2435925cfea1a0cdf42e7d55551013f8f7e4bc13e525e9021e1f022045e11c
Sep 09 03:41:38.579722 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:38.579680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs" event={"ID":"53de0404-d2b4-4814-8e32-b0f6b6c84ccb","Type":"ContainerStarted","Data":"9036c679be0fa47e977e65305244aaed18bd05e35a26540cf9ca3d8050e78f4b"}
Sep 09 03:41:38.580158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:38.579747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs" event={"ID":"53de0404-d2b4-4814-8e32-b0f6b6c84ccb","Type":"ContainerStarted","Data":"0f2435925cfea1a0cdf42e7d55551013f8f7e4bc13e525e9021e1f022045e11c"}
Sep 09 03:41:38.580158 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:38.579773    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:41:38.604081 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:38.604011    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs" podStartSLOduration=1.603988647 podStartE2EDuration="1.603988647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:41:38.602926556 +0000 UTC m=+2147.165253118" watchObservedRunningTime="2026-09-09 03:41:38.603988647 +0000 UTC m=+2147.166315193"
Sep 09 03:41:49.587489 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:41:49.587457    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-f5rgs"
Sep 09 03:42:14.809907 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.809872    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"]
Sep 09 03:42:14.810525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.810507    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy"
Sep 09 03:42:14.810605 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.810529    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bdddde-bd1c-4033-a7b2-fca98c7ae17d" containerName="istio-proxy"
Sep 09 03:42:14.814278 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.814255    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:14.816799 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.816772    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-52cf1b30-kserve-self-signed-certs\""
Sep 09 03:42:14.822384 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.822347    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"]
Sep 09 03:42:14.880220 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.880176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:14.923784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.923745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47sv\" (UniqueName: \"kubernetes.io/projected/a7172697-50e8-4725-8f09-553790214e1f-kube-api-access-x47sv\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:14.923784 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.923792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-dshm\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:14.924056 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.923888    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-model-cache\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:14.924056 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.923912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-tmp-dir\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:14.924056 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.923934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7172697-50e8-4725-8f09-553790214e1f-tls-certs\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:14.924056 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:14.923952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-home\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.024817 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.024772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7172697-50e8-4725-8f09-553790214e1f-tls-certs\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.024817 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.024817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-home\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.025092 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.024860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47sv\" (UniqueName: \"kubernetes.io/projected/a7172697-50e8-4725-8f09-553790214e1f-kube-api-access-x47sv\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.025092 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.024887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-dshm\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.025092 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.024966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-model-cache\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.025092 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.024988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-tmp-dir\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.025339 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.025311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-home\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.025390 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.025377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-tmp-dir\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.025431 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.025411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-model-cache\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.027507 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.027477    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-dshm\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.027654 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.027595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7172697-50e8-4725-8f09-553790214e1f-tls-certs\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.040462 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.040435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47sv\" (UniqueName: \"kubernetes.io/projected/a7172697-50e8-4725-8f09-553790214e1f-kube-api-access-x47sv\") pod \"e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") " pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.125940 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.125829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:15.410310 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.410282    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"]
Sep 09 03:42:15.410984 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:42:15.410958    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7172697_50e8_4725_8f09_553790214e1f.slice/crio-e38931210d717cb1aa87924ebf8ae517098923c6f6d51efd6c04639734f63faa WatchSource:0}: Error finding container e38931210d717cb1aa87924ebf8ae517098923c6f6d51efd6c04639734f63faa: Status 404 returned error can't find the container with id e38931210d717cb1aa87924ebf8ae517098923c6f6d51efd6c04639734f63faa
Sep 09 03:42:15.799185 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.799144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7" event={"ID":"a7172697-50e8-4725-8f09-553790214e1f","Type":"ContainerStarted","Data":"e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00"}
Sep 09 03:42:15.799350 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.799199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7" event={"ID":"a7172697-50e8-4725-8f09-553790214e1f","Type":"ContainerStarted","Data":"e38931210d717cb1aa87924ebf8ae517098923c6f6d51efd6c04639734f63faa"}
Sep 09 03:42:15.815708 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:15.815662    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7" podStartSLOduration=1.815646628 podStartE2EDuration="1.815646628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:42:15.814775403 +0000 UTC m=+2184.377101961" watchObservedRunningTime="2026-09-09 03:42:15.815646628 +0000 UTC m=+2184.377973173"
Sep 09 03:42:16.999154 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:16.999115    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"]
Sep 09 03:42:17.003034 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.003013    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.006028 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.006003    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-52cf1b30-kserve-self-signed-certs\""
Sep 09 03:42:17.017377 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.017352    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"]
Sep 09 03:42:17.043738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.043623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-home\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.043738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.043683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8251cda-e6c7-4064-b75b-11b0b3764a65-tls-certs\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.043738 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.043709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-dshm\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.044011 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.043888    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-tmp-dir\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.044011 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.043947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f2l\" (UniqueName: \"kubernetes.io/projected/c8251cda-e6c7-4064-b75b-11b0b3764a65-kube-api-access-s8f2l\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.044011 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.043980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-model-cache\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.067595 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.067535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:17.145245 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.145196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-tmp-dir\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.145446 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.145265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f2l\" (UniqueName: \"kubernetes.io/projected/c8251cda-e6c7-4064-b75b-11b0b3764a65-kube-api-access-s8f2l\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.145446 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.145312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-model-cache\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.145446 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.145388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-home\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.145446 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.145412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8251cda-e6c7-4064-b75b-11b0b3764a65-tls-certs\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.145710 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.145456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-dshm\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.145768 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.145699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-tmp-dir\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.146015 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.145985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-model-cache\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.146150 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.146030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-home\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.148392 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.148358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8251cda-e6c7-4064-b75b-11b0b3764a65-tls-certs\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.148484 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.148406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-dshm\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.157713 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.157692    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f2l\" (UniqueName: \"kubernetes.io/projected/c8251cda-e6c7-4064-b75b-11b0b3764a65-kube-api-access-s8f2l\") pod \"e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") " pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.312607 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.312535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:17.590404 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.590380    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"]
Sep 09 03:42:17.592741 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:42:17.592714    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8251cda_e6c7_4064_b75b_11b0b3764a65.slice/crio-fe3dcb2938c235f9b8fae719614c2a489ed94a7784ce564aa671aff709070fda WatchSource:0}: Error finding container fe3dcb2938c235f9b8fae719614c2a489ed94a7784ce564aa671aff709070fda: Status 404 returned error can't find the container with id fe3dcb2938c235f9b8fae719614c2a489ed94a7784ce564aa671aff709070fda
Sep 09 03:42:17.814893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.814851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8" event={"ID":"c8251cda-e6c7-4064-b75b-11b0b3764a65","Type":"ContainerStarted","Data":"e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8"}
Sep 09 03:42:17.814893 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.814896    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8" event={"ID":"c8251cda-e6c7-4064-b75b-11b0b3764a65","Type":"ContainerStarted","Data":"fe3dcb2938c235f9b8fae719614c2a489ed94a7784ce564aa671aff709070fda"}
Sep 09 03:42:17.832950 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:17.832901    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8" podStartSLOduration=1.832884463 podStartE2EDuration="1.832884463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:42:17.830235034 +0000 UTC m=+2186.392561574" watchObservedRunningTime="2026-09-09 03:42:17.832884463 +0000 UTC m=+2186.395211009"
Sep 09 03:42:18.644032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:18.643993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:25.126873 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:25.126835    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:25.127266 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:25.126894    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:25.139334 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:25.139311    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:25.870289 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:25.870256    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:27.313525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:27.313493    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:27.313525 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:27.313535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:27.326396 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:27.326366    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:27.881213 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:27.881183    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:39.124790 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:39.124756    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"]
Sep 09 03:42:39.142925 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:39.142820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:40.650665 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:40.650633    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"]
Sep 09 03:42:40.683227 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:40.683192    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:54.145810 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.145763    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7" podUID="a7172697-50e8-4725-8f09-553790214e1f" containerName="main" containerID="cri-o://e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00" gracePeriod=45
Sep 09 03:42:54.401031 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.400968    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:54.506066 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506034    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-tmp-dir\") pod \"a7172697-50e8-4725-8f09-553790214e1f\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") "
Sep 09 03:42:54.506280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506092    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x47sv\" (UniqueName: \"kubernetes.io/projected/a7172697-50e8-4725-8f09-553790214e1f-kube-api-access-x47sv\") pod \"a7172697-50e8-4725-8f09-553790214e1f\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") "
Sep 09 03:42:54.506280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7172697-50e8-4725-8f09-553790214e1f-tls-certs\") pod \"a7172697-50e8-4725-8f09-553790214e1f\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") "
Sep 09 03:42:54.506280 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-dshm\") pod \"a7172697-50e8-4725-8f09-553790214e1f\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") "
Sep 09 03:42:54.506472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506297    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7172697-50e8-4725-8f09-553790214e1f" (UID: "a7172697-50e8-4725-8f09-553790214e1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:54.506472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506349    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-home\") pod \"a7172697-50e8-4725-8f09-553790214e1f\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") "
Sep 09 03:42:54.506472 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506398    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-model-cache\") pod \"a7172697-50e8-4725-8f09-553790214e1f\" (UID: \"a7172697-50e8-4725-8f09-553790214e1f\") "
Sep 09 03:42:54.506699 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506668    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-home" (OuterVolumeSpecName: "home") pod "a7172697-50e8-4725-8f09-553790214e1f" (UID: "a7172697-50e8-4725-8f09-553790214e1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:54.506771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506722    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:54.506771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.506756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7172697-50e8-4725-8f09-553790214e1f" (UID: "a7172697-50e8-4725-8f09-553790214e1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:54.508468 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.508447    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7172697-50e8-4725-8f09-553790214e1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7172697-50e8-4725-8f09-553790214e1f" (UID: "a7172697-50e8-4725-8f09-553790214e1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:42:54.508904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.508885    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-dshm" (OuterVolumeSpecName: "dshm") pod "a7172697-50e8-4725-8f09-553790214e1f" (UID: "a7172697-50e8-4725-8f09-553790214e1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:54.508904 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.508893    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7172697-50e8-4725-8f09-553790214e1f-kube-api-access-x47sv" (OuterVolumeSpecName: "kube-api-access-x47sv") pod "a7172697-50e8-4725-8f09-553790214e1f" (UID: "a7172697-50e8-4725-8f09-553790214e1f"). InnerVolumeSpecName "kube-api-access-x47sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:42:54.607128 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.607101    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47sv\" (UniqueName: \"kubernetes.io/projected/a7172697-50e8-4725-8f09-553790214e1f-kube-api-access-x47sv\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:54.607128 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.607126    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7172697-50e8-4725-8f09-553790214e1f-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:54.607128 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.607136    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:54.607327 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.607143    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:54.607327 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:54.607151    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7172697-50e8-4725-8f09-553790214e1f-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:55.029206 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.029171    2562 generic.go:356] "Generic (PLEG): container finished" podID="a7172697-50e8-4725-8f09-553790214e1f" containerID="e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00" exitCode=0
Sep 09 03:42:55.029395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.029223    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"
Sep 09 03:42:55.029395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.029223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7" event={"ID":"a7172697-50e8-4725-8f09-553790214e1f","Type":"ContainerDied","Data":"e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00"}
Sep 09 03:42:55.029395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.029337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7" event={"ID":"a7172697-50e8-4725-8f09-553790214e1f","Type":"ContainerDied","Data":"e38931210d717cb1aa87924ebf8ae517098923c6f6d51efd6c04639734f63faa"}
Sep 09 03:42:55.029395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.029362    2562 scope.go:117] "RemoveContainer" containerID="e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00"
Sep 09 03:42:55.039704 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.039688    2562 scope.go:117] "RemoveContainer" containerID="e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00"
Sep 09 03:42:55.039982 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:42:55.039967    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00\": container with ID starting with e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00 not found: ID does not exist" containerID="e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00"
Sep 09 03:42:55.040030 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.039991    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00"} err="failed to get container status \"e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00\": rpc error: code = NotFound desc = could not find container \"e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00\": container with ID starting with e2ffe02dafe24beb639f977d8811eaff8a45747957e287813aa3fbe91be37e00 not found: ID does not exist"
Sep 09 03:42:55.050560 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.050518    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"]
Sep 09 03:42:55.054341 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.054313    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-52cf1b30-kserve-76df8c857f-7tvq7"]
Sep 09 03:42:55.672521 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.672454    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8" podUID="c8251cda-e6c7-4064-b75b-11b0b3764a65" containerName="main" containerID="cri-o://e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8" gracePeriod=45
Sep 09 03:42:55.928297 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.928238    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:55.942695 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:55.942649    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7172697-50e8-4725-8f09-553790214e1f" path="/var/lib/kubelet/pods/a7172697-50e8-4725-8f09-553790214e1f/volumes"
Sep 09 03:42:56.020602 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.020565    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-tmp-dir\") pod \"c8251cda-e6c7-4064-b75b-11b0b3764a65\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") "
Sep 09 03:42:56.020783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.020643    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8f2l\" (UniqueName: \"kubernetes.io/projected/c8251cda-e6c7-4064-b75b-11b0b3764a65-kube-api-access-s8f2l\") pod \"c8251cda-e6c7-4064-b75b-11b0b3764a65\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") "
Sep 09 03:42:56.020783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.020669    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-model-cache\") pod \"c8251cda-e6c7-4064-b75b-11b0b3764a65\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") "
Sep 09 03:42:56.020783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.020719    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8251cda-e6c7-4064-b75b-11b0b3764a65-tls-certs\") pod \"c8251cda-e6c7-4064-b75b-11b0b3764a65\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") "
Sep 09 03:42:56.020783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.020737    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-dshm\") pod \"c8251cda-e6c7-4064-b75b-11b0b3764a65\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") "
Sep 09 03:42:56.020783 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.020766    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-home\") pod \"c8251cda-e6c7-4064-b75b-11b0b3764a65\" (UID: \"c8251cda-e6c7-4064-b75b-11b0b3764a65\") "
Sep 09 03:42:56.021352 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.021240    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-home" (OuterVolumeSpecName: "home") pod "c8251cda-e6c7-4064-b75b-11b0b3764a65" (UID: "c8251cda-e6c7-4064-b75b-11b0b3764a65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:56.021352 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.021252    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8251cda-e6c7-4064-b75b-11b0b3764a65" (UID: "c8251cda-e6c7-4064-b75b-11b0b3764a65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:56.021352 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.021326    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8251cda-e6c7-4064-b75b-11b0b3764a65" (UID: "c8251cda-e6c7-4064-b75b-11b0b3764a65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:56.023276 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.023248    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8251cda-e6c7-4064-b75b-11b0b3764a65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8251cda-e6c7-4064-b75b-11b0b3764a65" (UID: "c8251cda-e6c7-4064-b75b-11b0b3764a65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:42:56.023364 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.023279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-dshm" (OuterVolumeSpecName: "dshm") pod "c8251cda-e6c7-4064-b75b-11b0b3764a65" (UID: "c8251cda-e6c7-4064-b75b-11b0b3764a65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:56.023364 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.023337    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8251cda-e6c7-4064-b75b-11b0b3764a65-kube-api-access-s8f2l" (OuterVolumeSpecName: "kube-api-access-s8f2l") pod "c8251cda-e6c7-4064-b75b-11b0b3764a65" (UID: "c8251cda-e6c7-4064-b75b-11b0b3764a65"). InnerVolumeSpecName "kube-api-access-s8f2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:42:56.039564 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.039518    2562 generic.go:356] "Generic (PLEG): container finished" podID="c8251cda-e6c7-4064-b75b-11b0b3764a65" containerID="e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8" exitCode=0
Sep 09 03:42:56.039667 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.039581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8" event={"ID":"c8251cda-e6c7-4064-b75b-11b0b3764a65","Type":"ContainerDied","Data":"e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8"}
Sep 09 03:42:56.039667 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.039598    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"
Sep 09 03:42:56.039667 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.039619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8" event={"ID":"c8251cda-e6c7-4064-b75b-11b0b3764a65","Type":"ContainerDied","Data":"fe3dcb2938c235f9b8fae719614c2a489ed94a7784ce564aa671aff709070fda"}
Sep 09 03:42:56.039667 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.039636    2562 scope.go:117] "RemoveContainer" containerID="e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8"
Sep 09 03:42:56.050894 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.050880    2562 scope.go:117] "RemoveContainer" containerID="e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8"
Sep 09 03:42:56.051167 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:42:56.051147    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8\": container with ID starting with e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8 not found: ID does not exist" containerID="e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8"
Sep 09 03:42:56.051231 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.051177    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8"} err="failed to get container status \"e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8\": rpc error: code = NotFound desc = could not find container \"e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8\": container with ID starting with e600500ab43298a089a28f917d39214f91dbb651bd1309b77c1c2fe43fa16fd8 not found: ID does not exist"
Sep 09 03:42:56.060674 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.060653    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"]
Sep 09 03:42:56.065208 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.065180    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-52cf1b30-kserve-7ddd6c459f-bdmc8"]
Sep 09 03:42:56.121611 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.121563    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8251cda-e6c7-4064-b75b-11b0b3764a65-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:56.121611 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.121601    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:56.121611 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.121611    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:56.121611 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.121619    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:56.121611 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.121630    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8f2l\" (UniqueName: \"kubernetes.io/projected/c8251cda-e6c7-4064-b75b-11b0b3764a65-kube-api-access-s8f2l\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:56.121933 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:56.121640    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8251cda-e6c7-4064-b75b-11b0b3764a65-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:42:57.939245 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:57.939204    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8251cda-e6c7-4064-b75b-11b0b3764a65" path="/var/lib/kubelet/pods/c8251cda-e6c7-4064-b75b-11b0b3764a65/volumes"
Sep 09 03:42:58.853391 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.853358    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76695d568-2wxxf"]
Sep 09 03:42:58.853812 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.853796    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7172697-50e8-4725-8f09-553790214e1f" containerName="main"
Sep 09 03:42:58.853871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.853814    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7172697-50e8-4725-8f09-553790214e1f" containerName="main"
Sep 09 03:42:58.853871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.853824    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8251cda-e6c7-4064-b75b-11b0b3764a65" containerName="main"
Sep 09 03:42:58.853871 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.853830    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8251cda-e6c7-4064-b75b-11b0b3764a65" containerName="main"
Sep 09 03:42:58.853973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.853911    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8251cda-e6c7-4064-b75b-11b0b3764a65" containerName="main"
Sep 09 03:42:58.853973 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.853921    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7172697-50e8-4725-8f09-553790214e1f" containerName="main"
Sep 09 03:42:58.858385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.858365    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:58.868880 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.868851    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76695d568-2wxxf"]
Sep 09 03:42:58.947772 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.947728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a9dbed-def3-47ab-850e-f2b5c8c82c87-maas-api-tls\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:58.948183 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.947792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a9dbed-def3-47ab-850e-f2b5c8c82c87-metrics-certs\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:58.948183 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.947823    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcft\" (UniqueName: \"kubernetes.io/projected/02a9dbed-def3-47ab-850e-f2b5c8c82c87-kube-api-access-mwcft\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:58.948183 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:58.947846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a9dbed-def3-47ab-850e-f2b5c8c82c87-openshift-service-ca\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.049212 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.049175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a9dbed-def3-47ab-850e-f2b5c8c82c87-maas-api-tls\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.049392 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.049227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a9dbed-def3-47ab-850e-f2b5c8c82c87-metrics-certs\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.049392 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.049352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcft\" (UniqueName: \"kubernetes.io/projected/02a9dbed-def3-47ab-850e-f2b5c8c82c87-kube-api-access-mwcft\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.049480 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.049395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a9dbed-def3-47ab-850e-f2b5c8c82c87-openshift-service-ca\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.050095 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.050073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a9dbed-def3-47ab-850e-f2b5c8c82c87-openshift-service-ca\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.052115 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.052086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a9dbed-def3-47ab-850e-f2b5c8c82c87-maas-api-tls\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.052293 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.052097    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a9dbed-def3-47ab-850e-f2b5c8c82c87-metrics-certs\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.057622 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.057599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcft\" (UniqueName: \"kubernetes.io/projected/02a9dbed-def3-47ab-850e-f2b5c8c82c87-kube-api-access-mwcft\") pod \"maas-api-76695d568-2wxxf\" (UID: \"02a9dbed-def3-47ab-850e-f2b5c8c82c87\") " pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.170958 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.170849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:42:59.447944 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.447903    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76695d568-2wxxf"]
Sep 09 03:42:59.448335 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:42:59.448306    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a9dbed_def3_47ab_850e_f2b5c8c82c87.slice/crio-2536468258dddaa582a1f49c71e35c868b00e5610c92121eed5c22d56684bc3f WatchSource:0}: Error finding container 2536468258dddaa582a1f49c71e35c868b00e5610c92121eed5c22d56684bc3f: Status 404 returned error can't find the container with id 2536468258dddaa582a1f49c71e35c868b00e5610c92121eed5c22d56684bc3f
Sep 09 03:42:59.449797 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:42:59.449771    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:43:00.066305 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:00.066272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf" event={"ID":"02a9dbed-def3-47ab-850e-f2b5c8c82c87","Type":"ContainerStarted","Data":"6a397e14f59f9a15bc135d114882b1d9bc4d859cb12c68fdd8591d3bc25378cd"}
Sep 09 03:43:00.066718 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:00.066329    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf" event={"ID":"02a9dbed-def3-47ab-850e-f2b5c8c82c87","Type":"ContainerStarted","Data":"2536468258dddaa582a1f49c71e35c868b00e5610c92121eed5c22d56684bc3f"}
Sep 09 03:43:00.066718 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:00.066351    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:43:00.081736 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:00.081671    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf" podStartSLOduration=1.5468314300000001 podStartE2EDuration="2.081647821s" totalImagesPullingTime="534.816391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:43:00.081322111 +0000 UTC m=+2228.643648681" watchObservedRunningTime="2026-09-09 03:43:00.081647821 +0000 UTC m=+2228.643974367"
Sep 09 03:43:06.078462 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:06.078431    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76695d568-2wxxf"
Sep 09 03:43:13.611864 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:13.611766    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"]
Sep 09 03:43:13.612313 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:13.612063    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8" podUID="d6873e2b-79ae-4648-b8fc-dd569f4bc207" containerName="maas-api" containerID="cri-o://072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d" gracePeriod=30
Sep 09 03:43:14.870422 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:14.870397    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:43:15.002608 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.002474    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6873e2b-79ae-4648-b8fc-dd569f4bc207-openshift-service-ca\") pod \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") "
Sep 09 03:43:15.002608 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.002516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlfcc\" (UniqueName: \"kubernetes.io/projected/d6873e2b-79ae-4648-b8fc-dd569f4bc207-kube-api-access-hlfcc\") pod \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") "
Sep 09 03:43:15.002860 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.002649    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-metrics-certs\") pod \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") "
Sep 09 03:43:15.002860 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.002675    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-maas-api-tls\") pod \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\" (UID: \"d6873e2b-79ae-4648-b8fc-dd569f4bc207\") "
Sep 09 03:43:15.002978 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.002913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6873e2b-79ae-4648-b8fc-dd569f4bc207-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6873e2b-79ae-4648-b8fc-dd569f4bc207" (UID: "d6873e2b-79ae-4648-b8fc-dd569f4bc207"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:43:15.003034 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.003003    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6873e2b-79ae-4648-b8fc-dd569f4bc207-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:43:15.005066 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.005039    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6873e2b-79ae-4648-b8fc-dd569f4bc207" (UID: "d6873e2b-79ae-4648-b8fc-dd569f4bc207"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:43:15.005162 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.005101    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6873e2b-79ae-4648-b8fc-dd569f4bc207" (UID: "d6873e2b-79ae-4648-b8fc-dd569f4bc207"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:43:15.005162 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.005133    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6873e2b-79ae-4648-b8fc-dd569f4bc207-kube-api-access-hlfcc" (OuterVolumeSpecName: "kube-api-access-hlfcc") pod "d6873e2b-79ae-4648-b8fc-dd569f4bc207" (UID: "d6873e2b-79ae-4648-b8fc-dd569f4bc207"). InnerVolumeSpecName "kube-api-access-hlfcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:43:15.103589 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.103501    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:43:15.103589 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.103531    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6873e2b-79ae-4648-b8fc-dd569f4bc207-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:43:15.103589 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.103578    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlfcc\" (UniqueName: \"kubernetes.io/projected/d6873e2b-79ae-4648-b8fc-dd569f4bc207-kube-api-access-hlfcc\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\""
Sep 09 03:43:15.158936 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.158895    2562 generic.go:356] "Generic (PLEG): container finished" podID="d6873e2b-79ae-4648-b8fc-dd569f4bc207" containerID="072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d" exitCode=0
Sep 09 03:43:15.159096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.158951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8" event={"ID":"d6873e2b-79ae-4648-b8fc-dd569f4bc207","Type":"ContainerDied","Data":"072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d"}
Sep 09 03:43:15.159096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.158963    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"
Sep 09 03:43:15.159096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.158997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8" event={"ID":"d6873e2b-79ae-4648-b8fc-dd569f4bc207","Type":"ContainerDied","Data":"24c25410309aea136897c279a348c7bacbc91f18da23586a05c30e773da8d346"}
Sep 09 03:43:15.159096 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.159022    2562 scope.go:117] "RemoveContainer" containerID="072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d"
Sep 09 03:43:15.169442 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.169423    2562 scope.go:117] "RemoveContainer" containerID="072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d"
Sep 09 03:43:15.169800 ip-10-0-143-52 kubenswrapper[2562]: E0909 03:43:15.169768    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d\": container with ID starting with 072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d not found: ID does not exist" containerID="072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d"
Sep 09 03:43:15.169903 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.169809    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d"} err="failed to get container status \"072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d\": rpc error: code = NotFound desc = could not find container \"072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d\": container with ID starting with 072f3cd41e79cae9bc4cef3e5a4284d591f0fd88c028b1e76f2c399d89e8502d not found: ID does not exist"
Sep 09 03:43:15.182110 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.182080    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"]
Sep 09 03:43:15.186976 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.186946    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-jd2s8"]
Sep 09 03:43:15.938781 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:15.938750    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6873e2b-79ae-4648-b8fc-dd569f4bc207" path="/var/lib/kubelet/pods/d6873e2b-79ae-4648-b8fc-dd569f4bc207/volumes"
Sep 09 03:43:23.921238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:23.921193    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:38.840447 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:38.840407    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-759nz_d4f3a691-d396-4ace-8db8-bc69767f5bb4/manager/0.log"
Sep 09 03:43:38.968430 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:38.968394    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-x5vf8_de70f2e3-d26d-47b1-9f70-d0a1daf67b25/manager/0.log"
Sep 09 03:43:39.485863 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:39.485826    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-75hs5_fe4b17cd-7c22-4e8c-8310-980a7ca13c85/copy-manifests/0.log"
Sep 09 03:43:39.497726 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:39.497695    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-75hs5_fe4b17cd-7c22-4e8c-8310-980a7ca13c85/manager/0.log"
Sep 09 03:43:40.141449 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:40.141420    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76695d568-2wxxf_02a9dbed-def3-47ab-850e-f2b5c8c82c87/maas-api/0.log"
Sep 09 03:43:40.503684 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:40.503587    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ql9d6_81eff2b3-3e9a-4838-ab55-86a00eb3a3f4/postgres/0.log"
Sep 09 03:43:41.448643 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:41.448612    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5ngn8_a2804f10-5673-483a-b801-659f8f230dfe/manager/0.log"
Sep 09 03:43:41.566836 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:41.566808    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8zs4k_6d7aad25-ad18-4c7e-9c87-f07150f0f0c0/manager/0.log"
Sep 09 03:43:41.685751 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:41.685721    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-llm7l_bdba21fe-37b1-4d6e-8958-f1023571ab1d/kuadrant-console-plugin/0.log"
Sep 09 03:43:41.816073 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:41.816038    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-f5rgs_53de0404-d2b4-4814-8e32-b0f6b6c84ccb/manager/0.log"
Sep 09 03:43:42.081347 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:42.081259    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-x4xfd_fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d/manager/0.log"
Sep 09 03:43:42.719027 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:42.718995    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8d5f99fb-74wrc_9ac1f072-6090-4fd1-bb05-27a571187828/kube-auth-proxy/0.log"
Sep 09 03:43:42.960323 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:42.960290    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zvtkx_9142fd43-df2e-4dc5-b146-029f01d15c7a/istio-proxy/1.log"
Sep 09 03:43:43.321666 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:43.321626    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-794f5b986b-77gpk_7042025b-b615-4b6e-a06c-eda170b933bb/router/0.log"
Sep 09 03:43:43.810445 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:43.810412    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7x6lw_14d91c90-1905-4376-918a-03a200464429/main/0.log"
Sep 09 03:43:44.296550 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:44.296504    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-rpv6j_746a5f45-210b-4dde-87af-15ffd135ac60/main/0.log"
Sep 09 03:43:50.236790 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.236741    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nn8m/must-gather-vgqkh"]
Sep 09 03:43:50.238238 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.238204    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6873e2b-79ae-4648-b8fc-dd569f4bc207" containerName="maas-api"
Sep 09 03:43:50.238418 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.238405    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6873e2b-79ae-4648-b8fc-dd569f4bc207" containerName="maas-api"
Sep 09 03:43:50.238748 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.238730    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6873e2b-79ae-4648-b8fc-dd569f4bc207" containerName="maas-api"
Sep 09 03:43:50.242740 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.242712    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nn8m/must-gather-vgqkh"
Sep 09 03:43:50.244884 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.244853    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nn8m\"/\"openshift-service-ca.crt\""
Sep 09 03:43:50.245016 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.244895    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nn8m\"/\"default-dockercfg-k9ktg\""
Sep 09 03:43:50.245657 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.245635    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nn8m\"/\"kube-root-ca.crt\""
Sep 09 03:43:50.245792 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.245633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nn8m/must-gather-vgqkh"]
Sep 09 03:43:50.268460 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.268407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:50.313992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.313946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwlk\" (UniqueName: \"kubernetes.io/projected/1e3213f9-f182-4f95-aee6-f0a7dd5b45c1-kube-api-access-vrwlk\") pod \"must-gather-vgqkh\" (UID: \"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1\") " pod="openshift-must-gather-9nn8m/must-gather-vgqkh"
Sep 09 03:43:50.313992 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.313990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e3213f9-f182-4f95-aee6-f0a7dd5b45c1-must-gather-output\") pod \"must-gather-vgqkh\" (UID: \"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1\") " pod="openshift-must-gather-9nn8m/must-gather-vgqkh"
Sep 09 03:43:50.336718 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.336683    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:50.415328 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.415284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwlk\" (UniqueName: \"kubernetes.io/projected/1e3213f9-f182-4f95-aee6-f0a7dd5b45c1-kube-api-access-vrwlk\") pod \"must-gather-vgqkh\" (UID: \"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1\") " pod="openshift-must-gather-9nn8m/must-gather-vgqkh"
Sep 09 03:43:50.415585 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.415332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e3213f9-f182-4f95-aee6-f0a7dd5b45c1-must-gather-output\") pod \"must-gather-vgqkh\" (UID: \"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1\") " pod="openshift-must-gather-9nn8m/must-gather-vgqkh"
Sep 09 03:43:50.415771 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.415749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e3213f9-f182-4f95-aee6-f0a7dd5b45c1-must-gather-output\") pod \"must-gather-vgqkh\" (UID: \"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1\") " pod="openshift-must-gather-9nn8m/must-gather-vgqkh"
Sep 09 03:43:50.423846 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.423821    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwlk\" (UniqueName: \"kubernetes.io/projected/1e3213f9-f182-4f95-aee6-f0a7dd5b45c1-kube-api-access-vrwlk\") pod \"must-gather-vgqkh\" (UID: \"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1\") " pod="openshift-must-gather-9nn8m/must-gather-vgqkh"
Sep 09 03:43:50.554086 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.554053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nn8m/must-gather-vgqkh"
Sep 09 03:43:50.849584 ip-10-0-143-52 kubenswrapper[2562]: W0909 03:43:50.849528    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e3213f9_f182_4f95_aee6_f0a7dd5b45c1.slice/crio-0a88df6c25a463994802644bf71725d717f143fcafa96a066ac544d9e3904e6c WatchSource:0}: Error finding container 0a88df6c25a463994802644bf71725d717f143fcafa96a066ac544d9e3904e6c: Status 404 returned error can't find the container with id 0a88df6c25a463994802644bf71725d717f143fcafa96a066ac544d9e3904e6c
Sep 09 03:43:50.850997 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:50.850957    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nn8m/must-gather-vgqkh"]
Sep 09 03:43:51.369533 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:51.369492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/must-gather-vgqkh" event={"ID":"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1","Type":"ContainerStarted","Data":"0a88df6c25a463994802644bf71725d717f143fcafa96a066ac544d9e3904e6c"}
Sep 09 03:43:52.381100 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:52.381059    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/must-gather-vgqkh" event={"ID":"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1","Type":"ContainerStarted","Data":"ae14adcbdfb7fbb0d5218f4aa980c72d56fef16df68aa37b3841ecc2ef782e73"}
Sep 09 03:43:52.381696 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:52.381662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/must-gather-vgqkh" event={"ID":"1e3213f9-f182-4f95-aee6-f0a7dd5b45c1","Type":"ContainerStarted","Data":"50b8f5805a828b3a2e790a67c666be1eae94501a673e266f46e74a1d72646f2e"}
Sep 09 03:43:52.401413 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:52.401338    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nn8m/must-gather-vgqkh" podStartSLOduration=1.613709453 podStartE2EDuration="2.401315333s" totalImagesPullingTime="787.60588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:43:52.398032385 +0000 UTC m=+2280.960358931" watchObservedRunningTime="2026-09-09 03:43:52.401315333 +0000 UTC m=+2280.963641879"
Sep 09 03:43:53.221976 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:53.221929    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4drfs_8a2c4bfb-6825-4173-93a0-96a27cbdaedd/global-pull-secret-syncer/0.log"
Sep 09 03:43:53.355554 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:53.355502    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log"
Sep 09 03:43:53.491932 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:53.491847    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-52.ec2.internal_c6639989942a5da5f614950f4219a4b4/haproxy/0.log"
Sep 09 03:43:58.423795 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:58.423761    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5ngn8_a2804f10-5673-483a-b801-659f8f230dfe/manager/0.log"
Sep 09 03:43:58.455834 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:58.455804    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8zs4k_6d7aad25-ad18-4c7e-9c87-f07150f0f0c0/manager/0.log"
Sep 09 03:43:58.483493 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:58.483463    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-llm7l_bdba21fe-37b1-4d6e-8958-f1023571ab1d/kuadrant-console-plugin/0.log"
Sep 09 03:43:58.546868 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:58.546815    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-f5rgs_53de0404-d2b4-4814-8e32-b0f6b6c84ccb/manager/0.log"
Sep 09 03:43:58.659395 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:43:58.659317    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-x4xfd_fb0b58c2-6582-4cd1-90e3-b61b9fbfdd3d/manager/0.log"
Sep 09 03:44:00.676610 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:00.676576    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrfff_9e7661a2-2851-40c7-8f24-f7901b78847b/node-exporter/0.log"
Sep 09 03:44:00.715963 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:00.715862    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrfff_9e7661a2-2851-40c7-8f24-f7901b78847b/kube-rbac-proxy/0.log"
Sep 09 03:44:00.744786 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:00.744756    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrfff_9e7661a2-2851-40c7-8f24-f7901b78847b/init-textfile/0.log"
Sep 09 03:44:00.792796 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:00.792767    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-58dll_877cf394-5003-4c55-89de-088237996d57/kube-rbac-proxy-main/0.log"
Sep 09 03:44:00.822558 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:00.822516    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-58dll_877cf394-5003-4c55-89de-088237996d57/kube-rbac-proxy-self/0.log"
Sep 09 03:44:00.854747 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:00.854714    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-58dll_877cf394-5003-4c55-89de-088237996d57/openshift-state-metrics/0.log"
Sep 09 03:44:01.242693 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:01.242642    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-845787b854-c79nh_c9f4522a-4fda-4785-9d24-0657ef72835b/thanos-query/0.log"
Sep 09 03:44:01.265569 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:01.265502    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-845787b854-c79nh_c9f4522a-4fda-4785-9d24-0657ef72835b/kube-rbac-proxy-web/0.log"
Sep 09 03:44:01.291032 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:01.290996    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-845787b854-c79nh_c9f4522a-4fda-4785-9d24-0657ef72835b/kube-rbac-proxy/0.log"
Sep 09 03:44:01.316897 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:01.316866    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-845787b854-c79nh_c9f4522a-4fda-4785-9d24-0657ef72835b/prom-label-proxy/0.log"
Sep 09 03:44:01.342302 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:01.342258    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-845787b854-c79nh_c9f4522a-4fda-4785-9d24-0657ef72835b/kube-rbac-proxy-rules/0.log"
Sep 09 03:44:01.365137 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:01.365108    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-845787b854-c79nh_c9f4522a-4fda-4785-9d24-0657ef72835b/kube-rbac-proxy-metrics/0.log"
Sep 09 03:44:02.811075 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:02.811041    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-4t4zd_4366ede7-855d-48f2-8285-e1fcf0e925ec/networking-console-plugin/0.log"
Sep 09 03:44:03.945736 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:03.945703    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"]
Sep 09 03:44:03.952385 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:03.952345    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:03.958568 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:03.958518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"]
Sep 09 03:44:03.991594 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:03.991564    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d9578b6c7-zmgw2_489fcd11-027c-445e-a133-1d61ad2dfca7/console/0.log"
Sep 09 03:44:04.061888 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.061845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-sys\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.062937 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.062907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-lib-modules\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.063449 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.063430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-proc\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.063680 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.063661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-podres\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.063830 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.063813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjcn\" (UniqueName: \"kubernetes.io/projected/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-kube-api-access-xxjcn\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.165339 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.165297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-proc\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.165612 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.165572    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-proc\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.165817 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.165793    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-podres\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.165910 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.165647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-podres\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.165910 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.165850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjcn\" (UniqueName: \"kubernetes.io/projected/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-kube-api-access-xxjcn\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.166014 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.165944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-sys\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.166014 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.165992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-lib-modules\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.166149 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.166127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-lib-modules\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.180527 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.169694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-sys\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.184284 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.184247    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjcn\" (UniqueName: \"kubernetes.io/projected/ca28c86c-6eda-4b6c-a89a-eafc0648b2fa-kube-api-access-xxjcn\") pod \"perf-node-gather-daemonset-nl2zk\" (UID: \"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.272121 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.272012    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:04.572486 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:04.572445    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"]
Sep 09 03:44:05.495616 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:05.495572    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk" event={"ID":"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa","Type":"ContainerStarted","Data":"6a9dac26b4ac5342535add34b21c2c0e936f3259751fd3625f21d5967e2e3acc"}
Sep 09 03:44:05.496281 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:05.496249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk" event={"ID":"ca28c86c-6eda-4b6c-a89a-eafc0648b2fa","Type":"ContainerStarted","Data":"96267b78c20649ef714c2d1797f90017839093049ed258dfc22b4957b433116a"}
Sep 09 03:44:05.496671 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:05.496650    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:05.514024 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:05.513953    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk" podStartSLOduration=2.513929431 podStartE2EDuration="2.513929431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:44:05.511758936 +0000 UTC m=+2294.074085482" watchObservedRunningTime="2026-09-09 03:44:05.513929431 +0000 UTC m=+2294.076255979"
Sep 09 03:44:05.802999 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:05.802964    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d466q_bbb59c21-963b-4254-8d19-d2fb97b78882/dns/0.log"
Sep 09 03:44:05.826008 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:05.825963    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d466q_bbb59c21-963b-4254-8d19-d2fb97b78882/kube-rbac-proxy/0.log"
Sep 09 03:44:06.001497 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:06.001459    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lplwr_226d6682-de69-414a-bec1-1a3114c89844/dns-node-resolver/0.log"
Sep 09 03:44:06.748906 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:06.748868    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwtvs_781959c0-527d-4192-86a4-99be89ec07ab/node-ca/0.log"
Sep 09 03:44:07.938792 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:07.938749    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8d5f99fb-74wrc_9ac1f072-6090-4fd1-bb05-27a571187828/kube-auth-proxy/0.log"
Sep 09 03:44:07.994749 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:07.994675    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zvtkx_9142fd43-df2e-4dc5-b146-029f01d15c7a/istio-proxy/0.log"
Sep 09 03:44:08.006220 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:08.006181    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zvtkx_9142fd43-df2e-4dc5-b146-029f01d15c7a/istio-proxy/1.log"
Sep 09 03:44:08.100986 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:08.100959    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-794f5b986b-77gpk_7042025b-b615-4b6e-a06c-eda170b933bb/router/0.log"
Sep 09 03:44:08.969590 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:08.969559    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6n2dq_be59ed35-76ae-464d-b7ff-5c3ded0d76c5/serve-healthcheck-canary/0.log"
Sep 09 03:44:09.518804 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:09.518775    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ggsw_d0855e8e-3df7-41a5-8d2d-eaad6ff17404/kube-rbac-proxy/0.log"
Sep 09 03:44:09.540397 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:09.540366    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ggsw_d0855e8e-3df7-41a5-8d2d-eaad6ff17404/exporter/0.log"
Sep 09 03:44:09.563956 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:09.563927    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ggsw_d0855e8e-3df7-41a5-8d2d-eaad6ff17404/extractor/0.log"
Sep 09 03:44:11.515987 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:11.515955    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-nl2zk"
Sep 09 03:44:11.815342 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:11.815313    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-759nz_d4f3a691-d396-4ace-8db8-bc69767f5bb4/manager/0.log"
Sep 09 03:44:12.059788 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:12.059752    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-x5vf8_de70f2e3-d26d-47b1-9f70-d0a1daf67b25/manager/0.log"
Sep 09 03:44:12.287779 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:12.287741    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-75hs5_fe4b17cd-7c22-4e8c-8310-980a7ca13c85/manager/0.log"
Sep 09 03:44:12.304529 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:12.304501    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-75hs5_fe4b17cd-7c22-4e8c-8310-980a7ca13c85/copy-manifests/0.log"
Sep 09 03:44:15.146753 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:15.146714    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8d5f99fb-74wrc_9ac1f072-6090-4fd1-bb05-27a571187828/kube-auth-proxy/0.log"
Sep 09 03:44:15.184099 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:15.184058    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zvtkx_9142fd43-df2e-4dc5-b146-029f01d15c7a/istio-proxy/0.log"
Sep 09 03:44:15.194846 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:15.194807    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zvtkx_9142fd43-df2e-4dc5-b146-029f01d15c7a/istio-proxy/1.log"
Sep 09 03:44:15.261254 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:15.261211    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-794f5b986b-77gpk_7042025b-b615-4b6e-a06c-eda170b933bb/router/0.log"
Sep 09 03:44:17.197426 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:17.197315    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8d5f99fb-74wrc_9ac1f072-6090-4fd1-bb05-27a571187828/kube-auth-proxy/0.log"
Sep 09 03:44:17.243872 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:17.243838    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zvtkx_9142fd43-df2e-4dc5-b146-029f01d15c7a/istio-proxy/0.log"
Sep 09 03:44:17.258581 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:17.258522    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zvtkx_9142fd43-df2e-4dc5-b146-029f01d15c7a/istio-proxy/1.log"
Sep 09 03:44:17.321010 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:17.320977    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-794f5b986b-77gpk_7042025b-b615-4b6e-a06c-eda170b933bb/router/0.log"
Sep 09 03:44:20.863941 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:20.863902    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q765z_ef19fc26-1f19-47bb-9d7d-3602e7c55f01/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:20.947905 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:20.947878    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgn7l_f2dbc178-5c81-4110-be6e-82b349c347fa/kube-multus-additional-cni-plugins/0.log"
Sep 09 03:44:20.968124 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:20.968094    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgn7l_f2dbc178-5c81-4110-be6e-82b349c347fa/egress-router-binary-copy/0.log"
Sep 09 03:44:20.987935 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:20.987900    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgn7l_f2dbc178-5c81-4110-be6e-82b349c347fa/cni-plugins/0.log"
Sep 09 03:44:21.010635 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:21.010609    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgn7l_f2dbc178-5c81-4110-be6e-82b349c347fa/bond-cni-plugin/0.log"
Sep 09 03:44:21.033080 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:21.033051    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgn7l_f2dbc178-5c81-4110-be6e-82b349c347fa/routeoverride-cni/0.log"
Sep 09 03:44:21.060647 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:21.060619    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgn7l_f2dbc178-5c81-4110-be6e-82b349c347fa/whereabouts-cni-bincopy/0.log"
Sep 09 03:44:21.084083 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:21.084053    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgn7l_f2dbc178-5c81-4110-be6e-82b349c347fa/whereabouts-cni/0.log"
Sep 09 03:44:21.217046 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:21.217011    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xr4dr_f799f4eb-9464-44af-aac1-ceb67e712d08/kube-multus/0.log"
Sep 09 03:44:21.281139 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:21.281110    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w75g9_c99b1fee-9619-4012-8bf1-19657cc1ac91/network-metrics-daemon/0.log"
Sep 09 03:44:21.297133 ip-10-0-143-52 kubenswrapper[2562]: I0909 03:44:21.297108    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w75g9_c99b1fee-9619-4012-8bf1-19657cc1ac91/kube-rbac-proxy/0.log"